Diffinery

codician/diffinerypublic

Open-source, agent-first GitHub and GitHub Actions alternative that runs on your own Cloudflare account: git hosting, pull requests, code review and drop-in GitHub Actions YAML on Workers, Durable Objects, R2 and Containers.

Container registry at diffinery.com, in place of ghcr.io (#16)

An OCI distribution registry at /v2/ with blobs in R2 and the index in D1. docker login works with personal tokens and the job GITHUB_TOKEN, whose packages permission decides push and pull. Packages are private by default and can be linked to a repository, which then shares its roles, like ghcr.io. A GitHub-shaped packages REST API and web pages list and manage them.

Hosted runners reach the registry through a local relay that splits layer uploads into chunks under Cloudflare's 100 MB request limit, for the Docker daemon and for buildx builders.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ruyx3Qr4jYrZEpPi466deb
Canberk Demirci authored · GitHub committed signed1 parent 7099aa3commit 31fe8fb572cc8488c8eb06301ecd6bdfeabb9005Browse files
24 files changed+3052−14
Files (24)
modified.github/workflows/live-test.yml+63−3View file
@@ -1,8 +1,9 @@
11 name: Live test
22
33 # End-to-end checks against the real diffinery.com after each deploy: hosted
4-# Actions jobs in Cloudflare Containers (with Docker builds and services), agent sandboxes, webhooks, scheduled
5-# workflows, Git LFS, and pull requests (test merge commits, pull_request_target and the merge queue). Each run works in a throwaway private repository that it
4+# Actions jobs in Cloudflare Containers (with Docker builds and services), the container registry, agent sandboxes,
5+# webhooks, scheduled workflows, Git LFS, and pull requests (test merge commits, pull_request_target and the merge
6+# queue). Each run works in a throwaway private repository that it
67 # deletes at the end.
78 #
89 # Needs the repository secret DIFFINERY_LIVE_TOKEN: a Diffinery token (Settings ->
@@ -23,7 +24,7 @@
2324
2425 jobs:
2526 live:
26- name: Containers, sandboxes, webhooks, schedules, LFS and pull requests
27+ name: Containers, registry, sandboxes, webhooks, schedules, LFS and pull requests
2728 # Runs on the host that deploys production (see the gate in deploy.yml).
2829 if: ${{ github.event_name == 'workflow_dispatch' || (github.server_url == 'https://diffinery.com' && github.event.workflow_run.conclusion == 'success') }}
2930 runs-on: ubuntu-latest
@@ -93,6 +94,59 @@
9394 printf 'PING\r\n' >&3
9495 read -r line <&3
9596 echo "service ok ${line%$'\r'}"
97+ # Container registry: a layer over Cloudflare's 100 MB request limit, pushed with
98+ # the job's GITHUB_TOKEN, plus the docker/*-action route most ghcr.io workflows use.
99+ publish:
100+ runs-on: ubuntu-latest
101+ permissions:
102+ packages: write
103+ steps:
104+ - run: |
105+ image="diffinery.com/${GITHUB_REPOSITORY,,}"
106+ echo "${{ '${{' }} secrets.GITHUB_TOKEN }}" | docker login diffinery.com -u "${{ '${{' }} github.actor }}" --password-stdin
107+ head -c 150000000 /dev/urandom > big.bin
108+ sha256sum big.bin | cut -d' ' -f1 > big.sha
109+ printf 'FROM scratch\nCOPY big.bin big.sha /\n' > Dockerfile
110+ docker build -q -t "$image:live" .
111+ docker push "$image:live"
112+ echo "pushed $(cat big.sha)"
113+ buildx:
114+ runs-on: ubuntu-latest
115+ permissions:
116+ packages: write
117+ steps:
118+ - run: |
119+ echo "IMAGE=diffinery.com/${GITHUB_REPOSITORY,,}" >> "$GITHUB_ENV"
120+ head -c 120000000 /dev/urandom > big.bin
121+ printf 'FROM scratch\nCOPY big.bin /\n' > Dockerfile
122+ - uses: docker/setup-buildx-action@v3
123+ - uses: docker/login-action@v3
124+ with:
125+ registry: diffinery.com
126+ username: ${{ '${{' }} github.actor }}
127+ password: ${{ '${{' }} secrets.GITHUB_TOKEN }}
128+ - uses: docker/build-push-action@v6
129+ with:
130+ context: .
131+ push: true
132+ tags: ${{ '${{' }} env.IMAGE }}:buildx
133+ pull:
134+ needs: [publish, buildx]
135+ runs-on: ubuntu-latest
136+ permissions:
137+ packages: read
138+ steps:
139+ - run: |
140+ image="diffinery.com/${GITHUB_REPOSITORY,,}"
141+ echo "${{ '${{' }} secrets.GITHUB_TOKEN }}" | docker login diffinery.com -u "${{ '${{' }} github.actor }}" --password-stdin
142+ docker pull -q "$image:live"
143+ id=$(docker create "$image:live" true)
144+ docker cp "$id:/big.bin" big.bin && docker cp "$id:/big.sha" big.sha
145+ [ "$(sha256sum big.bin | cut -d' ' -f1)" = "$(cat big.sha)" ] && echo "registry ok $(stat -c %s big.bin) bytes"
146+ docker pull -q "$image:buildx"
147+ id=$(docker create "$image:buildx" true)
148+ docker cp "$id:/big.bin" buildx.bin
149+ echo "buildx ok $(stat -c %s buildx.bin) bytes"
96150 EOF
97151 cat > .github/workflows/tick.yml <<'EOF'
98152 on:
@@ -130,6 +184,11 @@
130184 log=$(api "/repos/$OWNER/$REPO/actions/jobs/$job/logs")
131185 grep -F "docker ok built" <<<"$log" || { echo "::error::docker build or run failed on a hosted runner"; exit 1; }
132186 grep -F "service ok +PONG" <<<"$log" || { echo "::error::the redis service container was not reachable"; exit 1; }
187+ job=$(jq -r '.jobs[] | select(.name == "pull") | .id' <<<"$jobs")
188+ log=$(api "/repos/$OWNER/$REPO/actions/jobs/$job/logs")
189+ grep -F "registry ok 150000000 bytes" <<<"$log" || { echo "::error::the image pushed with a 150 MB layer did not come back intact from the registry"; exit 1; }
190+ grep -F "buildx ok 120000000 bytes" <<<"$log" || { echo "::error::the image pushed by docker/build-push-action did not come back from the registry"; exit 1; }
191+ api "/users/$OWNER/packages/container/${REPO,,}/versions" | jq -c '.[] | {tags: .metadata.container.tags, size, platforms}'
133192
134193 - name: Git LFS round trip
135194 if: steps.token.outputs.skip != 'true'
@@ -271,5 +330,6 @@
271330 - name: Delete the throwaway repositories
272331 if: always() && steps.token.outputs.skip != 'true'
273332 run: |
333+ curl -sS -o /dev/null -w 'package %{http_code}\n' -X DELETE "$DIFFINERY/api/v3/users/$OWNER/packages/container/${REPO,,}" -H "Authorization: Bearer $TOKEN"
274334 curl -sS -o /dev/null -w '%{http_code}\n' -X DELETE "$DIFFINERY/api/v3/repos/$OWNER/$REPO" -H "Authorization: Bearer $TOKEN"
275335 [ -z "${PR_REPO:-}" ] || curl -sS -o /dev/null -w '%{http_code}\n' -X DELETE "$DIFFINERY/api/v3/repos/$OWNER/$PR_REPO" -H "Authorization: Bearer $TOKEN"
modifiedREADME.md+6−1View file
@@ -27,7 +27,7 @@
2727 |---|---|
2828 | HTTP, git smart HTTP, REST API, MCP, web UI | Workers (Hono) |
2929 | One repository: refs, objects, PRs, issues, reviews, checks, runs | Durable Object with SQLite (`RepoDO`), one per repository |
30-| Large objects, Git LFS, logs, caches, artifacts, event archive | R2 |
30+| Large objects, Git LFS, container images, logs, caches, artifacts, event archive | R2 |
3131 | Accounts, orgs, tokens, invites, notifications, audit log | D1, read from nearby replicas through the Sessions API (each browser keeps a bookmark so it reads its own writes) |
3232 | Webhooks, notifications, audit fan-out | Queues (with a dead-letter queue) |
3333 | Workflow runs (durable, survive restarts, 6 h job timeout) | Workflows |
@@ -191,6 +191,11 @@
191191 their Dockerfile), and plain `docker build` / `docker run` / `docker compose`
192192 in steps. Hosted runners start the Docker daemon the first time a job uses
193193 it and pull Docker Hub images through `mirror.gcr.io`.
194+- A container registry at `diffinery.com` in place of ghcr.io: sign in with
195+ the job's `GITHUB_TOKEN` (`permissions: packages: write`) or a personal
196+ token, then push and pull `diffinery.com/<owner>/<name>`. Images are
197+ private by default and can be linked to a repository. See
198+ [docs/container-registry.md](docs/container-registry.md).
194199
195200 **Not supported yet:** `actions/github-script`,
196201 Windows and macOS hosted runners, and
addedapps/diffinery/migrations/0003_container_registry.sql+84−0View file
@@ -1,0 +1,84 @@
1+-- Container registry (OCI distribution API at /v2/). Blob and manifest bytes live in R2;
2+-- this is the index: packages, which blobs and manifests each one holds, tags, and uploads.
3+
4+-- A package is an image name below an owner: diffinery.com/<owner>/<name>.
5+-- Linked to a repository, it shares that repository's permissions (like ghcr.io).
6+CREATE TABLE packages (
7+ id TEXT PRIMARY KEY,
8+ owner_id TEXT NOT NULL REFERENCES accounts(id),
9+ name TEXT NOT NULL COLLATE NOCASE,
10+ repo_id TEXT REFERENCES repos(id) ON DELETE SET NULL,
11+ visibility TEXT NOT NULL DEFAULT 'private' CHECK (visibility IN ('private', 'internal', 'public')),
12+ description TEXT,
13+ created_by TEXT,
14+ created_at TEXT NOT NULL,
15+ updated_at TEXT NOT NULL,
16+ UNIQUE (owner_id, name)
17+);
18+CREATE INDEX packages_repo ON packages(repo_id);
19+
20+-- Content-addressed blobs, stored once however many packages use them.
21+CREATE TABLE registry_blobs (
22+ digest TEXT PRIMARY KEY,
23+ size INTEGER NOT NULL,
24+ r2_key TEXT NOT NULL,
25+ created_at TEXT NOT NULL
26+);
27+
28+CREATE TABLE package_blobs (
29+ package_id TEXT NOT NULL REFERENCES packages(id) ON DELETE CASCADE,
30+ digest TEXT NOT NULL,
31+ created_at TEXT NOT NULL,
32+ PRIMARY KEY (package_id, digest)
33+);
34+CREATE INDEX package_blobs_digest ON package_blobs(digest);
35+
36+CREATE TABLE package_manifests (
37+ package_id TEXT NOT NULL REFERENCES packages(id) ON DELETE CASCADE,
38+ digest TEXT NOT NULL,
39+ media_type TEXT NOT NULL,
40+ artifact_type TEXT,
41+ subject TEXT,
42+ annotations TEXT,
43+ size INTEGER NOT NULL,
44+ total_size INTEGER NOT NULL DEFAULT 0,
45+ platforms TEXT,
46+ pushed_by TEXT,
47+ created_at TEXT NOT NULL,
48+ PRIMARY KEY (package_id, digest)
49+);
50+CREATE INDEX package_manifests_subject ON package_manifests(package_id, subject);
51+CREATE INDEX package_manifests_digest ON package_manifests(digest);
52+
53+CREATE TABLE package_tags (
54+ package_id TEXT NOT NULL REFERENCES packages(id) ON DELETE CASCADE,
55+ tag TEXT NOT NULL,
56+ digest TEXT NOT NULL,
57+ updated_at TEXT NOT NULL,
58+ PRIMARY KEY (package_id, tag)
59+);
60+
61+-- Blob uploads in progress: an R2 object (small uploads) or multipart upload, plus a tail
62+-- object holding bytes that don't fill a whole part yet.
63+CREATE TABLE registry_uploads (
64+ id TEXT PRIMARY KEY,
65+ package_id TEXT NOT NULL REFERENCES packages(id) ON DELETE CASCADE,
66+ r2_key TEXT NOT NULL,
67+ multipart_id TEXT,
68+ parts TEXT NOT NULL DEFAULT '[]',
69+ size INTEGER NOT NULL DEFAULT 0,
70+ tail INTEGER NOT NULL DEFAULT 0,
71+ digest TEXT,
72+ created_at TEXT NOT NULL,
73+ updated_at TEXT NOT NULL
74+);
75+CREATE INDEX registry_uploads_updated ON registry_uploads(updated_at);
76+
77+-- Which blobs each manifest uses, so blobs a package no longer needs can be dropped.
78+CREATE TABLE package_manifest_blobs (
79+ package_id TEXT NOT NULL REFERENCES packages(id) ON DELETE CASCADE,
80+ manifest TEXT NOT NULL,
81+ digest TEXT NOT NULL,
82+ PRIMARY KEY (package_id, manifest, digest)
83+);
84+CREATE INDEX package_manifest_blobs_digest ON package_manifest_blobs(package_id, digest);
modifiedapps/diffinery/package.json+1−1View file
@@ -9,7 +9,7 @@
99 "typecheck": "tsc --noEmit && tsc --noEmit -p islands",
1010 "test": "vitest run --passWithNoTests --exclude 'test/e2e/**'",
1111 "cf-typegen": "wrangler types",
12- "e2e": "node --test --test-concurrency=1 --test-timeout=600000 test/e2e/e2e.test.mjs test/e2e/passkeys.test.mjs",
12+ "e2e": "node --test --test-concurrency=1 --test-timeout=600000 test/e2e/e2e.test.mjs test/e2e/passkeys.test.mjs test/e2e/registry.test.mjs",
1313 "db:migrate:local": "wrangler d1 migrations apply diffinery-dev --local -e dev",
1414 "build": "node islands/build.mjs",
1515 "predev": "node islands/build.mjs"
modifiedapps/diffinery/src/app.ts+10−1View file
@@ -13,6 +13,9 @@
1313 import { runner } from "./http/runner.js";
1414 import { api, wellKnown } from "./http/api.js";
1515 import { mcp } from "./mcp/server.js";
16+import { registry } from "./registry/http.js";
17+import { packagesApi } from "./registry/api.js";
18+import { RegistryStore } from "./registry/store.js";
1619
1720 export const app = new Hono<{ Bindings: Env; Variables: AppVars }>();
1821
@@ -60,6 +63,8 @@
6063 app.route("/", wellKnown);
6164 app.route("/", git);
6265 app.route("/", runner);
66+app.route("/", registry);
67+app.route("/", packagesApi);
6368 app.route("/", api);
6469 app.route("/", mcp);
6570
@@ -204,7 +209,11 @@
204209
205210 // ---------------------------------------------------------------- cron: workflow schedules
206211
207-export async function scheduled(controller: ScheduledController, env: Env): Promise<void> {
212+export async function scheduled(controller: ScheduledController, env: Env, ctx?: ExecutionContext): Promise<void> {
213+ // Container registry housekeeping: abandoned uploads and bytes no image uses any more.
214+ const cleanup = new RegistryStore(env).cleanup().catch((e) => console.error("registry cleanup failed", e));
215+ if (ctx) ctx.waitUntil(cleanup);
216+ else await cleanup;
208217 // The trigger runs every 5 minutes (GitHub's minimum schedule interval); fire anything due in the window.
209218 const end = new Date(controller.scheduledTime);
210219 end.setUTCSeconds(0, 0);
modifiedapps/diffinery/src/lib/util.ts+1−1View file
@@ -67,7 +67,7 @@
6767 }
6868
6969 /** Top-level paths the app serves; no account may use them as its login. */
70-export const RESERVED_LOGINS = new Set(["api", "mcp", "runner", "settings", "new", "notifications", "invite", "request-access", "healthz", "login", "logout", "admin", "explore", "favicon.ico", "robots.txt", "about", "docs", "static", "assets", "www", "git", "diffinery", "auth", "signin", "signup", "security", "-"]);
70+export const RESERVED_LOGINS = new Set(["api", "mcp", "runner", "settings", "new", "notifications", "invite", "request-access", "healthz", "login", "logout", "admin", "explore", "favicon.ico", "robots.txt", "about", "docs", "static", "assets", "www", "git", "diffinery", "auth", "signin", "signup", "security", "v2", "-"]);
7171
7272 export const isValidLogin = (s: string): boolean => isValidName(s) && !s.includes(".") && !RESERVED_LOGINS.has(s.toLowerCase());
7373
addedapps/diffinery/src/registry/access.ts+123−0View file
@@ -1,0 +1,123 @@
1+import type { AppContext } from "../auth/principal.js";
2+import type { Account, Repo, TokenScope } from "../lib/directory.js";
3+import { repoStub } from "../http/common.js";
4+import { atLeast, type Role } from "../repo/types.js";
5+import type { PackageRow } from "./store.js";
6+
7+/**
8+ * Who may do what with a container package. Mirrors ghcr.io:
9+ *
10+ * - A package linked to a repository shares its permissions: readers pull, writers push,
11+ * admins delete. The owner (the user, or the organization's owners and admins) always
12+ * administers its packages; organization members can push to unlinked ones.
13+ * - Packages are private unless made internal (any signed-in account pulls) or public.
14+ * - A workflow's GITHUB_TOKEN follows the job's `packages` permission. It pulls packages
15+ * of its own owner, pushes to its repository's packages (creating them, linked to the
16+ * repository, on first push) and deletes their versions.
17+ * - Personal tokens are capped by their scopes (read pulls, write pushes, admin deletes)
18+ * and by their repository list.
19+ */
20+export type PackageRole = "none" | "pull" | "push" | "admin";
21+
22+const RANK: Record<PackageRole, number> = { none: 0, pull: 1, push: 2, admin: 3 };
23+export const allows = (role: PackageRole, need: PackageRole) => RANK[role] >= RANK[need];
24+const max = (a: PackageRole, b: PackageRole): PackageRole => (RANK[a] >= RANK[b] ? a : b);
25+const min = (a: PackageRole, b: PackageRole): PackageRole => (RANK[a] <= RANK[b] ? a : b);
26+
27+const fromRepoRole = (r: Role | null): PackageRole => (!r ? "none" : r === "admin" ? "admin" : atLeast(r, "write") ? "push" : "pull");
28+const SCOPE_CAP: Record<TokenScope, PackageRole> = { read: "pull", write: "push", admin: "admin" };
29+
30+export interface PackageAccess {
31+ role: PackageRole;
32+ /** Repository a new package gets linked to when this caller creates it. */
33+ linkRepo: Repo | null;
34+ /** Who is acting, for audit and the "pushed by" column. */
35+ actor: { id: string; login: string } | null;
36+ /** Why a push or delete would be refused, for the error message. */
37+ hint?: string;
38+}
39+
40+export async function packageAccess(c: AppContext, owner: Account, name: string, pkg: PackageRow | null): Promise<PackageAccess> {
41+ const p = c.get("principal");
42+ const dir = c.get("dir");
43+
44+ if (p.type === "anonymous") return { role: pkg?.visibility === "public" ? "pull" : "none", linkRepo: null, actor: null };
45+
46+ if (p.type === "repo-token") {
47+ const repo = await dir.repoById(p.repoId);
48+ if (!repo) return { role: "none", linkRepo: null, actor: null };
49+ const stub = repoStub(c.env, repo.id);
50+ const auth = await stub.runnerAuth(p.token);
51+ if (!auth) return { role: "none", linkRepo: null, actor: null, hint: "the job token is no longer valid" };
52+ let perm: "none" | "read" | "write";
53+ let actor: PackageAccess["actor"];
54+ if (auth.kind === "job") {
55+ perm = await stub.packagesPermission(p.token);
56+ actor = { id: `job:${auth.jobId}`, login: "diffinery-actions[bot]" };
57+ } else {
58+ perm = atLeast(auth.caller.role, "write") ? "write" : "read";
59+ actor = { id: auth.caller.id, login: auth.caller.login };
60+ }
61+ const sameOwner = repo.owner_id === owner.id;
62+ let role: PackageRole = "none";
63+ let hint: string | undefined;
64+ if (pkg) {
65+ const linked = pkg.repo_id === repo.id;
66+ if (perm !== "none" && (linked || sameOwner || pkg.visibility !== "private")) role = "pull";
67+ if (perm === "write" && linked) role = "admin";
68+ else if (perm === "write" && sameOwner && !pkg.repo_id) role = "push";
69+ else if (pkg.repo_id && !linked) hint = `${owner.login}/${pkg.name} belongs to the ${owner.login}/${pkg.repo_name ?? "?"} repository, so only its workflows can push to it`;
70+ } else if (perm === "write" && sameOwner) role = "push";
71+ else if (!sameOwner) hint = `workflows in ${repo.owner_login}/${repo.name} can only create packages under ${repo.owner_login}`;
72+ if (perm === "none") hint = "the workflow's GITHUB_TOKEN has no packages permission; add `permissions: packages: write`";
73+ else if (perm === "read" && !hint) hint = "the workflow's GITHUB_TOKEN has packages: read; add `permissions: packages: write` to push";
74+ return { role, linkRepo: repo, actor, hint };
75+ }
76+
77+ const a = p.account;
78+ if (a.status !== "active") return { role: "none", linkRepo: null, actor: null };
79+ const actor = { id: a.id, login: a.login };
80+ const subject = a.kind === "agent" ? a.owner_id : a.id;
81+
82+ // The owner administers its packages; organization members push to unlinked ones.
83+ let role: PackageRole = "none";
84+ if (a.site_admin) role = "admin";
85+ else if (owner.kind === "user" && subject === owner.id) role = a.kind === "agent" ? "push" : "admin";
86+ else if (owner.kind === "org") {
87+ if (a.kind === "agent" && a.owner_id === owner.id) role = "push";
88+ else if (subject) {
89+ const r = await dir.orgRole(owner.id, subject);
90+ if (r === "owner" || r === "admin") role = a.kind === "agent" ? "push" : "admin";
91+ else if (r === "member") role = "push";
92+ }
93+ }
94+
95+ let linkRepo: Repo | null = null;
96+ if (pkg?.repo_id) {
97+ const repo = await dir.repoById(pkg.repo_id);
98+ if (repo) role = max(role, fromRepoRole(await dir.roleFor(a, repo)));
99+ } else if (!pkg) {
100+ // diffinery.com/<owner>/<repo>[/...] is linked to that repository when the pusher can write to it.
101+ const repo = await dir.repo(owner.login, name.split("/")[0]!);
102+ if (repo) {
103+ const r = fromRepoRole(await dir.roleFor(a, repo));
104+ if (allows(r, "push")) {
105+ linkRepo = repo;
106+ role = max(role, r);
107+ }
108+ }
109+ }
110+ if (pkg && (pkg.visibility === "public" || pkg.visibility === "internal")) role = max(role, "pull");
111+
112+ if (p.via === "token") {
113+ if (p.repoIds) {
114+ const rid = pkg ? pkg.repo_id : linkRepo?.id;
115+ if (!rid || !p.repoIds.includes(rid)) role = pkg && pkg.visibility !== "private" ? "pull" : "none";
116+ }
117+ const cap = p.scopes.reduce<PackageRole>((best, s) => max(best, SCOPE_CAP[s]), "none");
118+ role = min(role, cap);
119+ }
120+ let hint: string | undefined;
121+ if (p.via === "token" && !p.scopes.some((s) => s !== "read")) hint = "this token only has the read scope";
122+ return { role, linkRepo, actor, hint };
123+}
addedapps/diffinery/src/registry/api.ts+177−0View file
@@ -1,0 +1,177 @@
1+import { Hono } from "hono";
2+import type { Env } from "../env.js";
3+import type { AppContext, AppVars } from "../auth/principal.js";
4+import { requireAccount } from "../auth/principal.js";
5+import type { Account } from "../lib/directory.js";
6+import { HttpError } from "../lib/util.js";
7+import { loadRepo } from "../http/common.js";
8+import { allows, packageAccess, type PackageRole } from "./access.js";
9+import { RegistryStore, type ManifestRow, type PackageRow } from "./store.js";
10+
11+/**
12+ * Packages REST API, shaped like GitHub's (/user/packages, /orgs/{org}/packages,
13+ * versions and deletes), so scripts and actions such as
14+ * actions/delete-package-versions work. Only the container package type exists.
15+ */
16+export const packagesApi = new Hono<{ Bindings: Env; Variables: AppVars }>().basePath("/api/v3");
17+
18+const host = (c: AppContext) => new URL(c.env.PUBLIC_URL).host;
19+
20+export function packageJson(c: AppContext, p: PackageRow & { version_count?: number; tag_count?: number }) {
21+ const base = `${c.env.PUBLIC_URL}/${p.owner_login}/-/packages/container/${encodeURIComponent(p.name)}`;
22+ return {
23+ id: p.id,
24+ name: p.name,
25+ package_type: "container",
26+ owner: { login: p.owner_login, type: p.owner_kind === "org" ? "Organization" : "User" },
27+ version_count: p.version_count,
28+ visibility: p.visibility,
29+ description: p.description,
30+ image: `${host(c)}/${p.owner_login.toLowerCase()}/${p.name}`,
31+ url: `${c.env.PUBLIC_URL}/api/v3/${p.owner_kind === "org" ? "orgs" : "users"}/${p.owner_login}/packages/container/${encodeURIComponent(p.name)}`,
32+ html_url: base,
33+ repository: p.repo_id ? { id: p.repo_id, name: p.repo_name, full_name: `${p.owner_login}/${p.repo_name}`, html_url: `${c.env.PUBLIC_URL}/${p.owner_login}/${p.repo_name}` } : null,
34+ created_at: p.created_at,
35+ updated_at: p.updated_at,
36+ };
37+}
38+
39+export function versionJson(c: AppContext, p: PackageRow, v: ManifestRow & { tags: string[] }) {
40+ return {
41+ id: v.id,
42+ name: v.digest,
43+ url: `${packageJson(c, p).url}/versions/${v.id}`,
44+ package_html_url: packageJson(c, p).html_url,
45+ html_url: `${packageJson(c, p).html_url}/${v.digest}`,
46+ created_at: v.created_at,
47+ updated_at: v.created_at,
48+ media_type: v.media_type,
49+ size: v.total_size,
50+ platforms: v.platforms ? (JSON.parse(v.platforms) as Array<{ platform: string | null }>).map((x) => x.platform).filter(Boolean) : [],
51+ pushed_by: v.pushed_by,
52+ metadata: { package_type: "container", container: { tags: v.tags } },
53+ };
54+}
55+
56+function checkType(c: AppContext): void {
57+ const t = c.req.param("type");
58+ if (t && t !== "container" && t !== "docker") throw new HttpError(404, `Diffinery hosts container packages only (got ${t})`);
59+ const q = c.req.query("package_type");
60+ if (q && q !== "container" && q !== "docker") throw new HttpError(422, "package_type must be container");
61+}
62+
63+async function ownerAccount(c: AppContext, login: string, kind?: "org" | "user"): Promise<Account> {
64+ const a = await c.get("dir").accountByLogin(login);
65+ if (!a || a.kind === "agent" || (kind && a.kind !== kind)) throw new HttpError(404, "not found");
66+ return a;
67+}
68+
69+/** Packages under an owner that the caller can pull. */
70+async function listFor(c: AppContext, owner: Account) {
71+ checkType(c);
72+ const store = new RegistryStore(c.env);
73+ const rows = await store.packages({ ownerIds: [owner.id], all: false, limit: 1000 });
74+ const out = [];
75+ for (const p of rows.filter((r) => r.owner_id === owner.id)) {
76+ if (allows((await packageAccess(c, owner, p.name, p)).role, "pull")) out.push(packageJson(c, p));
77+ }
78+ const vis = c.req.query("visibility");
79+ return vis ? out.filter((p) => p.visibility === vis) : out;
80+}
81+
82+async function loadPackage(c: AppContext, owner: Account, need: PackageRole) {
83+ checkType(c);
84+ const name = decodeURIComponent(c.req.param("name")!).toLowerCase();
85+ const store = new RegistryStore(c.env);
86+ const pkg = await store.package(owner.id, name);
87+ if (!pkg) throw new HttpError(c.get("principal").type === "anonymous" ? 401 : 404, "package not found");
88+ const access = await packageAccess(c, owner, name, pkg);
89+ if (!allows(access.role, "pull")) throw new HttpError(c.get("principal").type === "anonymous" ? 401 : 404, "package not found");
90+ if (!allows(access.role, need)) throw new HttpError(403, `${need} access to this package is required${access.hint ? `: ${access.hint}` : ""}`);
91+ return { store, pkg, access };
92+}
93+
94+type OwnerOf = (c: AppContext) => Promise<Account>;
95+const owners: Array<[string, OwnerOf]> = [
96+ ["/user/packages", async (c) => requireAccount(c)],
97+ ["/users/:owner/packages", (c) => ownerAccount(c, c.req.param("owner")!, "user")],
98+ ["/orgs/:owner/packages", (c) => ownerAccount(c, c.req.param("owner")!, "org")],
99+];
100+
101+for (const [prefix, ownerOf] of owners) {
102+ packagesApi.get(prefix, async (c) => c.json(await listFor(c, await ownerOf(c))));
103+
104+ packagesApi.get(`${prefix}/:type/:name`, async (c) => {
105+ const owner = await ownerOf(c);
106+ const { store, pkg } = await loadPackage(c, owner, "pull");
107+ const versions = await store.versions(pkg.id);
108+ return c.json({ ...packageJson(c, pkg), version_count: versions.length });
109+ });
110+
111+ // Diffinery: change visibility or the linked repository ({"repository": "name"} or null).
112+ packagesApi.patch(`${prefix}/:type/:name`, async (c) => {
113+ const owner = await ownerOf(c);
114+ const { store, pkg } = await loadPackage(c, owner, "admin");
115+ const b = (await c.req.json().catch(() => ({}))) as { visibility?: string; repository?: string | null };
116+ const patch: Parameters<RegistryStore["updatePackage"]>[1] = {};
117+ if (b.visibility !== undefined) {
118+ if (!["private", "internal", "public"].includes(b.visibility)) throw new HttpError(422, "visibility must be private, internal or public");
119+ patch.visibility = b.visibility as PackageRow["visibility"];
120+ }
121+ if (b.repository !== undefined) {
122+ if (b.repository === null || b.repository === "") patch.repoId = null;
123+ else {
124+ const name = b.repository.includes("/") ? b.repository.split("/").pop()! : b.repository;
125+ const { repo } = await loadRepo(c, owner.login, name, "admin");
126+ patch.repoId = repo.id;
127+ }
128+ }
129+ await store.updatePackage(pkg.id, patch);
130+ await c.get("dir").audit({ actorId: requireAccount(c).id, actorLogin: requireAccount(c).login, action: "package.update", target: `${owner.login}/${pkg.name}`, detail: b });
131+ return c.json(packageJson(c, (await store.packageById(pkg.id))!));
132+ });
133+
134+ packagesApi.delete(`${prefix}/:type/:name`, async (c) => {
135+ const owner = await ownerOf(c);
136+ const { store, pkg, access } = await loadPackage(c, owner, "admin");
137+ await store.deletePackage(pkg.id);
138+ await c.get("dir").audit({ actorId: access.actor?.id, actorLogin: access.actor?.login, action: "package.delete", target: `${owner.login}/${pkg.name}` });
139+ return c.body(null, 204);
140+ });
141+
142+ packagesApi.get(`${prefix}/:type/:name/versions`, async (c) => {
143+ const owner = await ownerOf(c);
144+ const { store, pkg } = await loadPackage(c, owner, "pull");
145+ const perPage = Math.min(Math.max(Number(c.req.query("per_page") ?? 30) || 30, 1), 100);
146+ const page = Math.max(Number(c.req.query("page") ?? 1) || 1, 1);
147+ const tagged = c.req.query("state") === "tagged";
148+ let versions = await store.versions(pkg.id);
149+ if (tagged) versions = versions.filter((v) => v.tags.length);
150+ return c.json(versions.slice((page - 1) * perPage, page * perPage).map((v) => versionJson(c, pkg, v)));
151+ });
152+
153+ packagesApi.get(`${prefix}/:type/:name/versions/:id`, async (c) => {
154+ const owner = await ownerOf(c);
155+ const { store, pkg } = await loadPackage(c, owner, "pull");
156+ const v = await store.versionById(pkg.id, Number(c.req.param("id")));
157+ if (!v) throw new HttpError(404, "version not found");
158+ return c.json(versionJson(c, pkg, v));
159+ });
160+
161+ packagesApi.delete(`${prefix}/:type/:name/versions/:id`, async (c) => {
162+ const owner = await ownerOf(c);
163+ const { store, pkg, access } = await loadPackage(c, owner, "admin");
164+ const v = await store.versionById(pkg.id, Number(c.req.param("id")));
165+ if (!v) throw new HttpError(404, "version not found");
166+ await store.deleteManifest(pkg.id, v.digest);
167+ await c.get("dir").audit({ actorId: access.actor?.id, actorLogin: access.actor?.login, action: "package.delete_version", target: `${owner.login}/${pkg.name}@${v.digest}` });
168+ return c.body(null, 204);
169+ });
170+}
171+
172+// Diffinery: the packages linked to a repository.
173+packagesApi.get("/repos/:owner/:repo/packages", async (c) => {
174+ const { repo } = await loadRepo(c, c.req.param("owner"), c.req.param("repo"), "read");
175+ const rows = await new RegistryStore(c.env).packages({ repoId: repo.id });
176+ return c.json(rows.map((p) => packageJson(c, p)));
177+});
addedapps/diffinery/src/registry/http.ts+582−0View file
@@ -1,0 +1,582 @@
1+import { Hono } from "hono";
2+import type { Env } from "../env.js";
3+import type { AppContext, AppVars } from "../auth/principal.js";
4+import type { Account } from "../lib/directory.js";
5+import { HttpError } from "../lib/util.js";
6+import { allows, packageAccess, type PackageAccess, type PackageRole } from "./access.js";
7+import { DIGEST_RE, NAME_RE, RegistryStore, TAG_RE, manifestKey, type ManifestRow, type PackageRow, type UploadRow } from "./store.js";
8+import { UploadError, digestOf, discard, ingest } from "./upload.js";
9+
10+/**
11+ * Container registry: the OCI distribution API (what `docker login/push/pull`, buildx,
12+ * containerd, podman, crane, oras and helm speak) at https://diffinery.com/v2/.
13+ *
14+ * Images are named diffinery.com/<owner>/<name>, like ghcr.io/<owner>/<name>. Sign in
15+ * with any username and a Diffinery token as the password, or a workflow's GITHUB_TOKEN.
16+ */
17+export const registry = new Hono<{ Bindings: Env; Variables: AppVars }>();
18+
19+export const MANIFEST_TYPES = new Set([
20+ "application/vnd.oci.image.manifest.v1+json",
21+ "application/vnd.oci.image.index.v1+json",
22+ "application/vnd.docker.distribution.manifest.v2+json",
23+ "application/vnd.docker.distribution.manifest.list.v2+json",
24+]);
25+const INDEX_TYPES = new Set(["application/vnd.oci.image.index.v1+json", "application/vnd.docker.distribution.manifest.list.v2+json"]);
26+const MAX_MANIFEST = 4 * 1024 * 1024;
27+const API_HEADERS = { "docker-distribution-api-version": "registry/2.0" };
28+
29+export class RegistryError extends Error {
30+ constructor(
31+ public status: number,
32+ public code: string,
33+ message: string,
34+ public headers: Record<string, string> = {},
35+ ) {
36+ super(message);
37+ }
38+}
39+
40+function errorJson(status: number, code: string, message: string, headers: Record<string, string> = {}): Response {
41+ return new Response(JSON.stringify({ errors: [{ code, message, detail: null }] }), { status, headers: { "content-type": "application/json", ...API_HEADERS, ...headers } });
42+}
43+
44+const challenge = (c: AppContext) => ({ "www-authenticate": `Basic realm="${new URL(c.env.PUBLIC_URL).host}", charset="UTF-8"` });
45+
46+const hex = (b: ArrayBuffer) => [...new Uint8Array(b)].map((x) => x.toString(16).padStart(2, "0")).join("");
47+
48+// ---------------------------------------------------------------- routing
49+
50+registry.all("/v2", (c) => c.redirect("/v2/", 301));
51+
52+registry.all("/v2/*", async (c) => {
53+ try {
54+ const res = await route(c);
55+ for (const [k, v] of Object.entries(API_HEADERS)) res.headers.set(k, v);
56+ return res;
57+ } catch (e) {
58+ if (e instanceof RegistryError) return errorJson(e.status, e.code, e.message, e.status === 401 ? { ...challenge(c), ...e.headers } : e.headers);
59+ if (e instanceof UploadError) return errorJson(400, "BLOB_UPLOAD_INVALID", e.message);
60+ if (e instanceof HttpError) return errorJson(e.status, e.status === 401 ? "UNAUTHORIZED" : e.status === 403 ? "DENIED" : e.status === 404 ? "NAME_UNKNOWN" : "UNSUPPORTED", e.detail, e.status === 401 ? challenge(c) : {});
61+ console.error("registry error", c.req.method, c.req.path, e);
62+ return errorJson(500, "UNKNOWN", "internal error");
63+ }
64+});
65+
66+async function route(c: AppContext): Promise<Response> {
67+ const path = c.req.path;
68+ const method = c.req.method;
69+ if (path === "/v2/") {
70+ if (c.get("principal").type === "anonymous") throw new RegistryError(401, "UNAUTHORIZED", "sign in with a Diffinery token: docker login " + new URL(c.env.PUBLIC_URL).host);
71+ return c.json({});
72+ }
73+ if (path === "/v2/_catalog") return catalog(c);
74+ let m: RegExpExecArray | null;
75+ if ((m = /^\/v2\/(.+)\/tags\/list$/.exec(path))) return tagsList(c, m[1]!);
76+ if ((m = /^\/v2\/(.+)\/manifests\/([^/]+)$/.exec(path))) {
77+ if (method === "GET" || method === "HEAD") return getManifest(c, m[1]!, m[2]!);
78+ if (method === "PUT") return putManifest(c, m[1]!, m[2]!);
79+ if (method === "DELETE") return deleteManifest(c, m[1]!, m[2]!);
80+ }
81+ if ((m = /^\/v2\/(.+)\/referrers\/([^/]+)$/.exec(path)) && method === "GET") return referrers(c, m[1]!, m[2]!);
82+ if ((m = /^\/v2\/(.+)\/blobs\/uploads\/?$/.exec(path)) && method === "POST") return startUpload(c, m[1]!);
83+ if ((m = /^\/v2\/(.+)\/blobs\/uploads\/([0-9a-f]{32})$/.exec(path))) {
84+ if (method === "GET") return uploadStatus(c, m[1]!, m[2]!);
85+ if (method === "PATCH") return patchUpload(c, m[1]!, m[2]!);
86+ if (method === "PUT") return putUpload(c, m[1]!, m[2]!);
87+ if (method === "DELETE") return cancelUpload(c, m[1]!, m[2]!);
88+ }
89+ if ((m = /^\/v2\/(.+)\/blobs\/([^/]+)$/.exec(path))) {
90+ if (method === "GET" || method === "HEAD") return getBlob(c, m[1]!, m[2]!);
91+ if (method === "DELETE") return deleteBlob(c, m[1]!, m[2]!);
92+ }
93+ throw new RegistryError(404, "UNSUPPORTED", `${method} ${path} is not part of the registry API`);
94+}
95+
96+// ---------------------------------------------------------------- names and access
97+
98+interface Target {
99+ /** The name as the client wrote it (owner/name), for Location headers. */
100+ full: string;
101+ owner: Account;
102+ name: string;
103+ pkg: PackageRow | null;
104+ access: PackageAccess;
105+ store: RegistryStore;
106+}
107+
108+/** Splits <owner>/<name>, loads the package and checks the caller may `need` it. */
109+async function target(c: AppContext, full: string, need: PackageRole): Promise<Target> {
110+ const lower = full.toLowerCase();
111+ const slash = lower.indexOf("/");
112+ const name = slash > 0 ? lower.slice(slash + 1) : "";
113+ if (slash <= 0 || !NAME_RE.test(name) || lower.length > 255) {
114+ throw new RegistryError(400, "NAME_INVALID", `invalid image name "${full}": use ${new URL(c.env.PUBLIC_URL).host}/<owner>/<name> with lowercase letters, digits, '.', '_', '-' and '/'`);
115+ }
116+ const store = new RegistryStore(c.env);
117+ const owner = await c.get("dir").accountByLogin(lower.slice(0, slash));
118+ if (!owner || owner.kind === "agent") {
119+ if (c.get("principal").type === "anonymous") throw new RegistryError(401, "UNAUTHORIZED", "authentication required");
120+ throw new RegistryError(404, "NAME_UNKNOWN", `no user or organization named ${lower.slice(0, slash)}`);
121+ }
122+ const pkg = await store.package(owner.id, name);
123+ const access = await packageAccess(c, owner, name, pkg);
124+ if (!allows(access.role, need)) {
125+ const anon = c.get("principal").type === "anonymous";
126+ if (anon) throw new RegistryError(401, "UNAUTHORIZED", "authentication required");
127+ if (!allows(access.role, "pull")) {
128+ if (need === "pull" || pkg) throw new RegistryError(404, "NAME_UNKNOWN", `repository ${owner.login}/${name} not found${access.hint ? ` (${access.hint})` : ""}`);
129+ throw new RegistryError(403, "DENIED", `you can't create ${owner.login}/${name}${access.hint ? `: ${access.hint}` : ""}`);
130+ }
131+ throw new RegistryError(403, "DENIED", `${need === "admin" ? "deleting from" : "pushing to"} ${owner.login}/${name} is not allowed${access.hint ? `: ${access.hint}` : ""}`);
132+ }
133+ return { full, owner, name, pkg, access, store };
134+}
135+
136+/** The package, created on first push (linked to a repository when the access check found one). */
137+async function ensurePackage(c: AppContext, t: Target): Promise<PackageRow> {
138+ if (t.pkg) return t.pkg;
139+ const pkg = await t.store.createPackage(t.owner.id, t.name, t.access.linkRepo?.id ?? null, t.access.actor?.id ?? "unknown");
140+ c.executionCtx.waitUntil(
141+ c
142+ .get("dir")
143+ .audit({ actorId: t.access.actor?.id, actorLogin: t.access.actor?.login, action: "package.create", target: `${t.owner.login}/${t.name}`, detail: { repository: t.access.linkRepo ? `${t.access.linkRepo.owner_login}/${t.access.linkRepo.name}` : null } })
144+ .catch((e) => console.error("audit failed", e)),
145+ );
146+ t.pkg = pkg;
147+ return pkg;
148+}
149+
150+function requirePackage(t: Target): PackageRow {
151+ if (!t.pkg) throw new RegistryError(404, "NAME_UNKNOWN", `repository ${t.owner.login}/${t.name} not found`);
152+ return t.pkg;
153+}
154+
155+// ---------------------------------------------------------------- blobs
156+
157+async function getBlob(c: AppContext, full: string, digest: string): Promise<Response> {
158+ if (!DIGEST_RE.test(digest)) throw new RegistryError(400, "DIGEST_INVALID", "only sha256 digests are supported");
159+ const t = await target(c, full, "pull");
160+ const pkg = requirePackage(t);
161+ const blob = await t.store.packageBlob(pkg.id, digest);
162+ if (!blob) {
163+ // Manifests may be fetched through the blob endpoint too.
164+ const man = await t.store.manifest(pkg.id, digest);
165+ if (man) return manifestResponse(c, man, c.req.method === "HEAD");
166+ throw new RegistryError(404, "BLOB_UNKNOWN", `blob ${digest} not found`);
167+ }
168+ const headers: Record<string, string> = { "docker-content-digest": digest, "content-type": "application/octet-stream", etag: `"${digest}"`, "accept-ranges": "bytes", "cache-control": "private, max-age=31536000, immutable" };
169+ if (c.req.method === "HEAD") return new Response(null, { status: 200, headers: { ...headers, "content-length": String(blob.size) } });
170+ const range = parseRange(c.req.header("range"), blob.size);
171+ if (range === "invalid") return new Response(null, { status: 416, headers: { "content-range": `bytes */${blob.size}` } });
172+ const obj = await c.env.BUCKET.get(blob.r2_key, range ? { range: { offset: range.start, length: range.end - range.start + 1 } } : undefined);
173+ if (!obj) throw new RegistryError(404, "BLOB_UNKNOWN", `blob ${digest} is missing from storage`);
174+ if (range) return new Response(obj.body, { status: 206, headers: { ...headers, "content-length": String(range.end - range.start + 1), "content-range": `bytes ${range.start}-${range.end}/${blob.size}` } });
175+ return new Response(obj.body, { status: 200, headers: { ...headers, "content-length": String(blob.size) } });
176+}
177+
178+function parseRange(h: string | undefined, size: number): { start: number; end: number } | null | "invalid" {
179+ if (!h) return null;
180+ const m = /^bytes=(\d*)-(\d*)$/.exec(h.trim());
181+ if (!m || (!m[1] && !m[2])) return null;
182+ let start: number;
183+ let end: number;
184+ if (!m[1]) {
185+ start = Math.max(0, size - Number(m[2]));
186+ end = size - 1;
187+ } else {
188+ start = Number(m[1]);
189+ end = m[2] ? Math.min(Number(m[2]), size - 1) : size - 1;
190+ }
191+ if (start >= size || end < start) return "invalid";
192+ return { start, end };
193+}
194+
195+async function deleteBlob(c: AppContext, full: string, digest: string): Promise<Response> {
196+ if (!DIGEST_RE.test(digest)) throw new RegistryError(400, "DIGEST_INVALID", "only sha256 digests are supported");
197+ const t = await target(c, full, "admin");
198+ const pkg = requirePackage(t);
199+ if (!(await t.store.packageBlob(pkg.id, digest))) throw new RegistryError(404, "BLOB_UNKNOWN", `blob ${digest} not found`);
200+ await t.store.unlinkBlob(pkg.id, digest);
201+ return new Response(null, { status: 202 });
202+}
203+
204+const uploadUrl = (t: Target, id: string) => `/v2/${t.full}/blobs/uploads/${id}`;
205+
206+function uploadHeaders(t: Target, u: UploadRow): Record<string, string> {
207+ // Range can't tell an empty upload from a one-byte one, so the size is spelled out too.
208+ return { location: uploadUrl(t, u.id), range: `0-${Math.max(0, u.size - 1)}`, "docker-upload-uuid": u.id, "diffinery-upload-size": String(u.size), "content-length": "0" };
209+}
210+
211+async function startUpload(c: AppContext, full: string): Promise<Response> {
212+ const t = await target(c, full, "push");
213+ const pkg = await ensurePackage(c, t);
214+ const mount = c.req.query("mount");
215+ const from = c.req.query("from");
216+ if (mount && DIGEST_RE.test(mount)) {
217+ // Cross-repository mount: reuse a blob the caller can already pull from another package.
218+ if (await t.store.packageBlob(pkg.id, mount)) return blobCreated(t, mount);
219+ if (from) {
220+ try {
221+ const src = await target(c, from, "pull");
222+ if (src.pkg && (await t.store.packageBlob(src.pkg.id, mount))) {
223+ await t.store.linkBlob(pkg.id, mount);
224+ return blobCreated(t, mount);
225+ }
226+ } catch (e) {
227+ if (!(e instanceof RegistryError)) throw e;
228+ }
229+ }
230+ }
231+ const u = await t.store.createUpload(pkg.id);
232+ const digest = c.req.query("digest");
233+ if (digest) {
234+ // Monolithic upload: the whole blob in the POST body.
235+ if (!DIGEST_RE.test(digest)) throw new RegistryError(400, "DIGEST_INVALID", "only sha256 digests are supported");
236+ return finishUpload(c, t, pkg, await ingest(c.env, u, c.req.raw.body, true), digest);
237+ }
238+ return new Response(null, { status: 202, headers: uploadHeaders(t, u) });
239+}
240+
241+async function loadUpload(c: AppContext, full: string, id: string, need: PackageRole = "push"): Promise<{ t: Target; pkg: PackageRow; u: UploadRow }> {
242+ const t = await target(c, full, need);
243+ const pkg = requirePackage(t);
244+ const u = await t.store.upload(id);
245+ if (!u || u.package_id !== pkg.id) throw new RegistryError(404, "BLOB_UPLOAD_UNKNOWN", "upload not found; it may have expired");
246+ return { t, pkg, u };
247+}
248+
249+async function uploadStatus(c: AppContext, full: string, id: string): Promise<Response> {
250+ const { t, u } = await loadUpload(c, full, id);
251+ return new Response(null, { status: 204, headers: uploadHeaders(t, u) });
252+}
253+
254+function checkContentRange(c: AppContext, t: Target, u: UploadRow): void {
255+ const cr = c.req.header("content-range");
256+ if (!cr) return;
257+ const m = /^(?:bytes[ =])?(\d+)-(\d+)(?:\/\d+|\/\*)?$/.exec(cr.trim());
258+ if (!m || Number(m[1]) !== u.size) throw new RegistryError(416, "BLOB_UPLOAD_INVALID", `chunk must start at byte ${u.size}`, uploadHeaders(t, u));
259+}
260+
261+async function patchUpload(c: AppContext, full: string, id: string): Promise<Response> {
262+ const { t, u } = await loadUpload(c, full, id);
263+ checkContentRange(c, t, u);
264+ const next = await ingest(c.env, u, c.req.raw.body, false);
265+ await t.store.saveUpload(next);
266+ return new Response(null, { status: 202, headers: uploadHeaders(t, next) });
267+}
268+
269+async function putUpload(c: AppContext, full: string, id: string): Promise<Response> {
270+ const { t, pkg, u } = await loadUpload(c, full, id);
271+ const digest = c.req.query("digest");
272+ if (!digest || !DIGEST_RE.test(digest)) throw new RegistryError(400, "DIGEST_INVALID", "the digest query parameter must be a sha256 digest");
273+ if (c.req.raw.body) checkContentRange(c, t, u);
274+ return finishUpload(c, t, pkg, await ingest(c.env, u, c.req.raw.body, true), digest);
275+}
276+
277+async function cancelUpload(c: AppContext, full: string, id: string): Promise<Response> {
278+ const { t, u } = await loadUpload(c, full, id);
279+ await discard(c.env, u);
280+ await t.store.deleteUpload(u.id);
281+ return new Response(null, { status: 204 });
282+}
283+
284+/** Verifies a completed upload against its digest and adds it to the package. */
285+async function finishUpload(c: AppContext, t: Target, pkg: PackageRow, u: UploadRow, expected: string): Promise<Response> {
286+ let actual = u.digest;
287+ if (!actual) actual = (await digestOf(c.env, u.r2_key))?.digest ?? null;
288+ if (actual !== expected) {
289+ await discard(c.env, u);
290+ await t.store.deleteUpload(u.id);
291+ throw new RegistryError(400, "DIGEST_INVALID", `the uploaded content has digest ${actual ?? "unknown"}, not ${expected}`);
292+ }
293+ await t.store.addBlob(expected, u.size, u.r2_key);
294+ await t.store.linkBlob(pkg.id, expected);
295+ // Linked now, so cleanup can't drop the index row; if another upload already stored these bytes, ours go.
296+ const stored = await t.store.blob(expected);
297+ if (!stored) await t.store.addBlob(expected, u.size, u.r2_key);
298+ else if (stored.r2_key !== u.r2_key) await c.env.BUCKET.delete(u.r2_key);
299+ await t.store.deleteUpload(u.id);
300+ return blobCreated(t, expected);
301+}
302+
303+function blobCreated(t: Target, digest: string): Response {
304+ return new Response(null, { status: 201, headers: { location: `/v2/${t.full}/blobs/${digest}`, "docker-content-digest": digest, "content-length": "0" } });
305+}
306+
307+// ---------------------------------------------------------------- manifests
308+
309+async function resolveReference(t: Target, pkg: PackageRow, reference: string): Promise<ManifestRow | null> {
310+ if (DIGEST_RE.test(reference)) return t.store.manifest(pkg.id, reference);
311+ if (!TAG_RE.test(reference)) throw new RegistryError(400, "MANIFEST_INVALID", `invalid tag "${reference}"`);
312+ const digest = await t.store.tagDigest(pkg.id, reference);
313+ return digest ? t.store.manifest(pkg.id, digest) : null;
314+}
315+
316+async function manifestResponse(c: AppContext, m: ManifestRow, head: boolean): Promise<Response> {
317+ const headers: Record<string, string> = { "content-type": m.media_type, "docker-content-digest": m.digest, etag: `"${m.digest}"`, "content-length": String(m.size) };
318+ if (head) return new Response(null, { status: 200, headers });
319+ const obj = await c.env.BUCKET.get(manifestKey(m.digest));
320+ if (!obj) throw new RegistryError(404, "MANIFEST_UNKNOWN", `manifest ${m.digest} is missing from storage`);
321+ return new Response(obj.body, { status: 200, headers });
322+}
323+
324+async function getManifest(c: AppContext, full: string, reference: string): Promise<Response> {
325+ const t = await target(c, full, "pull");
326+ const pkg = requirePackage(t);
327+ const m = await resolveReference(t, pkg, reference);
328+ if (!m) throw new RegistryError(404, "MANIFEST_UNKNOWN", `manifest ${reference} not found in ${t.owner.login}/${t.name}`);
329+ return manifestResponse(c, m, c.req.method === "HEAD");
330+}
331+
332+interface Descriptor {
333+ mediaType?: string;
334+ digest?: string;
335+ size?: number;
336+ urls?: string[];
337+ artifactType?: string;
338+ annotations?: Record<string, string>;
339+ platform?: { os?: string; architecture?: string; variant?: string };
340+}
341+
342+interface ManifestJson {
343+ schemaVersion?: number;
344+ mediaType?: string;
345+ artifactType?: string;
346+ config?: Descriptor;
347+ layers?: Descriptor[];
348+ blobs?: Descriptor[];
349+ manifests?: Descriptor[];
350+ subject?: Descriptor;
351+ annotations?: Record<string, string>;
352+}
353+
354+const platformName = (p?: Descriptor["platform"]) => (p?.os && p.architecture ? `${p.os}/${p.architecture}${p.variant ? `/${p.variant}` : ""}` : null);
355+
356+async function putManifest(c: AppContext, full: string, reference: string): Promise<Response> {
357+ const t = await target(c, full, "push");
358+ const isDigest = DIGEST_RE.test(reference);
359+ if (!isDigest && !TAG_RE.test(reference)) throw new RegistryError(400, "MANIFEST_INVALID", `invalid tag "${reference}"`);
360+ const declared = Number(c.req.header("content-length") ?? 0);
361+ if (declared > MAX_MANIFEST) throw new RegistryError(413, "MANIFEST_INVALID", "manifests are limited to 4 MiB");
362+ const bytes = new Uint8Array(await c.req.arrayBuffer());
363+ if (bytes.length > MAX_MANIFEST) throw new RegistryError(413, "MANIFEST_INVALID", "manifests are limited to 4 MiB");
364+ const digest = `sha256:${hex(await crypto.subtle.digest("SHA-256", bytes))}`;
365+ if (isDigest && reference !== digest) throw new RegistryError(400, "DIGEST_INVALID", `the manifest's digest is ${digest}, not ${reference}`);
366+ let json: ManifestJson;
367+ try {
368+ json = JSON.parse(new TextDecoder().decode(bytes)) as ManifestJson;
369+ } catch {
370+ throw new RegistryError(400, "MANIFEST_INVALID", "the manifest is not valid JSON");
371+ }
372+ if (typeof json !== "object" || json === null || json.schemaVersion !== 2) throw new RegistryError(400, "MANIFEST_INVALID", "only schemaVersion 2 manifests (OCI or Docker v2) are supported");
373+ const header = (c.req.header("content-type") ?? "").split(";")[0]!.trim();
374+ const mediaType = json.mediaType || (MANIFEST_TYPES.has(header) ? header : json.manifests ? "application/vnd.oci.image.index.v1+json" : "application/vnd.oci.image.manifest.v1+json");
375+ if (!MANIFEST_TYPES.has(mediaType)) throw new RegistryError(400, "MANIFEST_INVALID", `unsupported manifest media type ${mediaType}`);
376+ if (json.mediaType && MANIFEST_TYPES.has(header) && header !== json.mediaType) throw new RegistryError(400, "MANIFEST_INVALID", `Content-Type ${header} does not match the manifest's mediaType ${json.mediaType}`);
377+
378+ const pkg = await ensurePackage(c, t);
379+ const isIndex = INDEX_TYPES.has(mediaType);
380+ let totalSize = 0;
381+ let platforms: Array<{ digest?: string; platform: string | null }> = [];
382+ const blobRefs: string[] = [];
383+ if (isIndex) {
384+ const children = (json.manifests ?? []).filter((d) => d.digest && DIGEST_RE.test(d.digest));
385+ if (children.length !== (json.manifests ?? []).length) throw new RegistryError(400, "MANIFEST_INVALID", "every entry in manifests needs a sha256 digest");
386+ const have = await t.store.manifestSizes(pkg.id, children.map((d) => d.digest!));
387+ const missing = children.find((d) => !have.has(d.digest!));
388+ if (missing) throw new RegistryError(400, "MANIFEST_BLOB_UNKNOWN", `manifest ${missing.digest} must be pushed before the index that lists it`);
389+ platforms = children.map((d) => ({ digest: d.digest, platform: platformName(d.platform) }));
390+ // What pulling every platform downloads; attestations and other artifacts don't count.
391+ totalSize = children.filter((d) => platformName(d.platform) !== "unknown/unknown").reduce((n, d) => n + (have.get(d.digest!) ?? 0), 0);
392+ } else {
393+ const descs = [...(json.config ? [json.config] : []), ...(json.layers ?? []), ...(json.blobs ?? [])];
394+ if (!json.config?.digest) throw new RegistryError(400, "MANIFEST_INVALID", "an image manifest needs a config descriptor");
395+ const local = descs.filter((d) => !d.urls?.length);
396+ for (const d of local) if (!d.digest || !DIGEST_RE.test(d.digest)) throw new RegistryError(400, "MANIFEST_INVALID", "every config and layer needs a sha256 digest");
397+ const digests = [...new Set(local.map((d) => d.digest!))];
398+ const have = await t.store.packageBlobs(pkg.id, digests);
399+ const missing = digests.find((d) => !have.has(d));
400+ if (missing) throw new RegistryError(400, "MANIFEST_BLOB_UNKNOWN", `blob ${missing} must be uploaded before the manifest that uses it`);
401+ blobRefs.push(...digests);
402+ totalSize = descs.reduce((n, d) => n + (d.size ?? 0), 0);
403+ const config = await readConfig(c, json.config);
404+ if (config) {
405+ platforms = [{ platform: platformName(config) }];
406+ await linkBySourceLabel(c, t, pkg, config.config?.Labels ?? {});
407+ const desc = config.config?.Labels?.["org.opencontainers.image.description"];
408+ if (desc && !pkg.description) await t.store.updatePackage(pkg.id, { description: desc.slice(0, 500) });
409+ }
410+ }
411+ const desc = json.annotations?.["org.opencontainers.image.description"];
412+ if (desc && !pkg.description) await t.store.updatePackage(pkg.id, { description: desc.slice(0, 500) });
413+ if (json.annotations?.["org.opencontainers.image.source"]) await linkBySourceLabel(c, t, pkg, json.annotations);
414+
415+ const key = manifestKey(digest);
416+ if (!(await c.env.BUCKET.head(key))) await c.env.BUCKET.put(key, bytes, { httpMetadata: { contentType: mediaType } });
417+ const subject = json.subject?.digest && DIGEST_RE.test(json.subject.digest) ? json.subject.digest : null;
418+ await t.store.putManifest(
419+ {
420+ package_id: pkg.id,
421+ digest,
422+ media_type: mediaType,
423+ artifact_type: json.artifactType ?? (json.config?.mediaType && json.config.mediaType !== "application/vnd.oci.image.config.v1+json" && json.config.mediaType !== "application/vnd.docker.container.image.v1+json" ? json.config.mediaType : null),
424+ subject,
425+ annotations: json.annotations ? JSON.stringify(json.annotations) : null,
426+ size: bytes.length,
427+ total_size: totalSize,
428+ platforms: platforms.length ? JSON.stringify(platforms) : null,
429+ pushed_by: t.access.actor?.login ?? null,
430+ created_at: new Date().toISOString(),
431+ },
432+ isDigest ? null : reference,
433+ blobRefs,
434+ );
435+ if (!isDigest) {
436+ c.executionCtx.waitUntil(
437+ c
438+ .get("dir")
439+ .audit({ actorId: t.access.actor?.id, actorLogin: t.access.actor?.login, action: "package.push", target: `${t.owner.login}/${t.name}:${reference}`, detail: { digest } })
440+ .catch((e) => console.error("audit failed", e)),
441+ );
442+ }
443+ const headers: Record<string, string> = { location: `/v2/${t.full}/manifests/${digest}`, "docker-content-digest": digest, "content-length": "0" };
444+ if (subject) headers["oci-subject"] = subject;
445+ return new Response(null, { status: 201, headers });
446+}
447+
448+interface ImageConfig {
449+ os?: string;
450+ architecture?: string;
451+ variant?: string;
452+ config?: { Labels?: Record<string, string> | null };
453+}
454+
455+/** The image config (platform and labels), when it's a small JSON blob in this registry. */
456+async function readConfig(c: AppContext, d: Descriptor): Promise<ImageConfig | null> {
457+ if (d.mediaType && d.mediaType !== "application/vnd.oci.image.config.v1+json" && d.mediaType !== "application/vnd.docker.container.image.v1+json") return null;
458+ if (!d.digest || (d.size ?? 0) > 1024 * 1024) return null;
459+ const blob = await new RegistryStore(c.env).blob(d.digest);
460+ if (!blob) return null;
461+ const obj = await c.env.BUCKET.get(blob.r2_key);
462+ if (!obj) return null;
463+ try {
464+ const cfg = (await obj.json()) as ImageConfig;
465+ return { os: cfg.os, architecture: cfg.architecture, variant: cfg.variant, config: { Labels: cfg.config?.Labels ?? {} } };
466+ } catch {
467+ return null;
468+ }
469+}
470+
471+/**
472+ * Links an unlinked package to the repository its org.opencontainers.image.source label
473+ * names (a Diffinery URL, or a GitHub URL of a repository moved here), when the pusher
474+ * can write to that repository. Like ghcr.io, the package then shares its permissions.
475+ */
476+async function linkBySourceLabel(c: AppContext, t: Target, pkg: PackageRow, labels: Record<string, string>): Promise<void> {
477+ if (pkg.repo_id) return;
478+ const src = labels["org.opencontainers.image.source"];
479+ if (!src) return;
480+ let url: URL;
481+ try {
482+ url = new URL(src);
483+ } catch {
484+ return;
485+ }
486+ const ours = new URL(c.env.PUBLIC_URL).host;
487+ if (url.host !== ours && url.host !== "github.com") return;
488+ const [owner, repoName] = url.pathname.replace(/^\/+/, "").replace(/\.git$/, "").split("/");
489+ if (!owner || !repoName || owner.toLowerCase() !== t.owner.login.toLowerCase()) return;
490+ const repo = await c.get("dir").repo(t.owner.login, repoName);
491+ if (!repo) return;
492+ const p = c.get("principal");
493+ let canWrite = false;
494+ if (p.type === "repo-token") canWrite = p.repoId === repo.id && allows(t.access.role, "push");
495+ else if (p.type === "account") {
496+ const r = await c.get("dir").roleFor(p.account, repo);
497+ canWrite = !!r && ["write", "maintain", "admin"].includes(r);
498+ }
499+ if (!canWrite) return;
500+ await t.store.updatePackage(pkg.id, { repoId: repo.id });
501+ pkg.repo_id = repo.id;
502+ pkg.repo_name = repo.name;
503+}
504+
505+async function deleteManifest(c: AppContext, full: string, reference: string): Promise<Response> {
506+ const t = await target(c, full, "admin");
507+ const pkg = requirePackage(t);
508+ if (DIGEST_RE.test(reference)) {
509+ if (!(await t.store.deleteManifest(pkg.id, reference))) throw new RegistryError(404, "MANIFEST_UNKNOWN", `manifest ${reference} not found`);
510+ } else {
511+ if (!TAG_RE.test(reference)) throw new RegistryError(400, "MANIFEST_INVALID", `invalid tag "${reference}"`);
512+ if (!(await t.store.deleteTag(pkg.id, reference))) throw new RegistryError(404, "MANIFEST_UNKNOWN", `tag ${reference} not found`);
513+ }
514+ c.executionCtx.waitUntil(
515+ c
516+ .get("dir")
517+ .audit({ actorId: t.access.actor?.id, actorLogin: t.access.actor?.login, action: "package.delete_version", target: `${t.owner.login}/${t.name}@${reference}` })
518+ .catch((e) => console.error("audit failed", e)),
519+ );
520+ return new Response(null, { status: 202 });
521+}
522+
523+async function referrers(c: AppContext, full: string, digest: string): Promise<Response> {
524+ if (!DIGEST_RE.test(digest)) throw new RegistryError(400, "DIGEST_INVALID", "only sha256 digests are supported");
525+ const t = await target(c, full, "pull");
526+ const pkg = requirePackage(t);
527+ const type = c.req.query("artifactType");
528+ let rows = await t.store.referrers(pkg.id, digest);
529+ if (type) rows = rows.filter((r) => r.artifact_type === type);
530+ const body = {
531+ schemaVersion: 2,
532+ mediaType: "application/vnd.oci.image.index.v1+json",
533+ manifests: rows.map((r) => ({ mediaType: r.media_type, digest: r.digest, size: r.size, ...(r.artifact_type ? { artifactType: r.artifact_type } : {}), ...(r.annotations ? { annotations: JSON.parse(r.annotations) } : {}) })),
534+ };
535+ const headers: Record<string, string> = { "content-type": "application/vnd.oci.image.index.v1+json" };
536+ if (type) headers["oci-filters-applied"] = "artifactType";
537+ return new Response(JSON.stringify(body), { status: 200, headers });
538+}
539+
540+// ---------------------------------------------------------------- listing
541+
542+async function tagsList(c: AppContext, full: string): Promise<Response> {
543+ const t = await target(c, full, "pull");
544+ const pkg = requirePackage(t);
545+ const nq = c.req.query("n");
546+ const n = nq === undefined ? undefined : Math.max(0, Math.min(Number(nq) || 0, 10000));
547+ const last = c.req.query("last");
548+ const tags = await t.store.tags(pkg.id, { n: n === undefined ? undefined : n + 1, last });
549+ const page = n === undefined ? tags : tags.slice(0, n);
550+ const headers: Record<string, string> = {};
551+ if (n !== undefined && tags.length > n && page.length) headers.link = `</v2/${t.full}/tags/list?n=${n}&last=${encodeURIComponent(page[page.length - 1]!)}>; rel="next"`;
552+ return c.json({ name: `${t.owner.login}/${t.name}`, tags: page }, 200, headers);
553+}
554+
555+async function catalog(c: AppContext): Promise<Response> {
556+ const p = c.get("principal");
557+ if (p.type === "anonymous") throw new RegistryError(401, "UNAUTHORIZED", "authentication required");
558+ const dir = c.get("dir");
559+ const store = new RegistryStore(c.env);
560+ let rows: PackageRow[];
561+ if (p.type === "account" && p.account.site_admin && !p.repoIds) rows = await store.packages({ all: true, limit: 5000 });
562+ else if (p.type === "account") {
563+ const subject = p.account.kind === "agent" ? p.account.owner_id! : p.account.id;
564+ const orgs = (await dir.orgsFor(subject)).map((o) => o.id);
565+ const repos = (await dir.reposVisibleTo(p.account)).map((r) => r.id);
566+ rows = await store.packages({ ownerIds: p.repoIds ? [] : [subject, ...orgs], repoIds: p.repoIds ?? repos, limit: 5000 });
567+ } else {
568+ const repo = await dir.repoById(p.repoId);
569+ rows = repo ? await store.packages({ ownerIds: [repo.owner_id], limit: 5000 }) : [];
570+ }
571+ const names = rows.map((r) => `${r.owner_login.toLowerCase()}/${r.name}`).sort();
572+ const nq = c.req.query("n");
573+ const last = c.req.query("last");
574+ let page = last ? names.filter((x) => x > last) : names;
575+ const headers: Record<string, string> = {};
576+ if (nq !== undefined) {
577+ const n = Math.max(0, Number(nq) || 0);
578+ if (page.length > n && n > 0) headers.link = `</v2/_catalog?n=${n}&last=${encodeURIComponent(page[n - 1]!)}>; rel="next"`;
579+ page = page.slice(0, n);
580+ }
581+ return c.json({ repositories: page }, 200, headers);
582+}
addedapps/diffinery/src/registry/store.ts+347−0View file
@@ -1,0 +1,347 @@
1+import type { Env } from "../env.js";
2+import { newId, now } from "../lib/util.js";
3+
4+/** A container image name below an owner: diffinery.com/<owner_login>/<name>. */
5+export interface PackageRow {
6+ id: string;
7+ owner_id: string;
8+ owner_login: string;
9+ owner_kind: "user" | "org" | "agent";
10+ name: string;
11+ repo_id: string | null;
12+ repo_name: string | null;
13+ visibility: "private" | "internal" | "public";
14+ description: string | null;
15+ created_by: string | null;
16+ created_at: string;
17+ updated_at: string;
18+}
19+
20+export interface ManifestRow {
21+ /** rowid, used as the numeric version id in the REST API. */
22+ id?: number;
23+ package_id: string;
24+ digest: string;
25+ media_type: string;
26+ artifact_type: string | null;
27+ subject: string | null;
28+ annotations: string | null;
29+ size: number;
30+ total_size: number;
31+ platforms: string | null;
32+ pushed_by: string | null;
33+ created_at: string;
34+}
35+
36+export interface UploadRow {
37+ id: string;
38+ package_id: string;
39+ r2_key: string;
40+ multipart_id: string | null;
41+ parts: string;
42+ size: number;
43+ tail: number;
44+ digest: string | null;
45+ created_at: string;
46+ updated_at: string;
47+}
48+
49+export interface BlobRow {
50+ digest: string;
51+ size: number;
52+ r2_key: string;
53+ created_at: string;
54+}
55+
56+export const DIGEST_RE = /^sha256:[a-f0-9]{64}$/;
57+export const TAG_RE = /^[A-Za-z0-9_][A-Za-z0-9._-]{0,127}$/;
58+/** OCI repository path components below the owner, lowercase only. */
59+export const NAME_RE = /^[a-z0-9]+(?:(?:\.|_|__|-+)[a-z0-9]+)*(?:\/[a-z0-9]+(?:(?:\.|_|__|-+)[a-z0-9]+)*)*$/;
60+
61+export const manifestKey = (digest: string) => `registry/manifests/${digest.replace(":", "/")}`;
62+export const blobKey = (uploadId: string) => `registry/blobs/${uploadId}`;
63+export const tailKey = (uploadId: string) => `registry/uploads/${uploadId}/tail`;
64+
65+const PKG_SELECT = "SELECT p.*, a.login AS owner_login, a.kind AS owner_kind, r.name AS repo_name FROM packages p JOIN accounts a ON a.id = p.owner_id LEFT JOIN repos r ON r.id = p.repo_id";
66+
67+/** The registry's index in D1. Bytes live in R2 under registry/. */
68+export class RegistryStore {
69+ constructor(private env: Env) {}
70+
71+ private get db() {
72+ return this.env.DB;
73+ }
74+
75+ // ---------------------------------------------------------------- packages
76+
77+ async package(ownerId: string, name: string): Promise<PackageRow | null> {
78+ return this.db.prepare(`${PKG_SELECT} WHERE p.owner_id = ? AND p.name = ?`).bind(ownerId, name).first<PackageRow>();
79+ }
80+
81+ async packageById(id: string): Promise<PackageRow | null> {
82+ return this.db.prepare(`${PKG_SELECT} WHERE p.id = ?`).bind(id).first<PackageRow>();
83+ }
84+
85+ /** Creates the package unless a concurrent push just did; returns it either way. */
86+ async createPackage(ownerId: string, name: string, repoId: string | null, createdBy: string): Promise<PackageRow> {
87+ const t = now();
88+ await this.db
89+ .prepare("INSERT INTO packages (id, owner_id, name, repo_id, visibility, created_by, created_at, updated_at) VALUES (?, ?, ?, ?, 'private', ?, ?, ?) ON CONFLICT(owner_id, name) DO NOTHING")
90+ .bind(newId(), ownerId, name, repoId, createdBy, t, t)
91+ .run();
92+ return (await this.package(ownerId, name))!;
93+ }
94+
95+ async packages(filter: { ownerIds?: string[]; repoIds?: string[]; repoId?: string; all?: boolean; limit?: number }): Promise<Array<PackageRow & { tag_count: number; version_count: number }>> {
96+ const where: string[] = [];
97+ const params: unknown[] = [];
98+ if (filter.repoId) {
99+ where.push("p.repo_id = ?");
100+ params.push(filter.repoId);
101+ } else if (!filter.all) {
102+ const ors: string[] = [];
103+ if (filter.ownerIds?.length) {
104+ ors.push(`p.owner_id IN (${filter.ownerIds.map(() => "?").join(",")})`);
105+ params.push(...filter.ownerIds);
106+ }
107+ if (filter.repoIds?.length) {
108+ ors.push(`p.repo_id IN (${filter.repoIds.map(() => "?").join(",")})`);
109+ params.push(...filter.repoIds);
110+ }
111+ ors.push("p.visibility != 'private'");
112+ where.push(`(${ors.join(" OR ")})`);
113+ }
114+ const q = `SELECT p.*, a.login AS owner_login, a.kind AS owner_kind, r.name AS repo_name,
115+ (SELECT count(*) FROM package_tags t WHERE t.package_id = p.id) AS tag_count,
116+ (SELECT count(*) FROM package_manifests m WHERE m.package_id = p.id) AS version_count
117+ FROM packages p JOIN accounts a ON a.id = p.owner_id LEFT JOIN repos r ON r.id = p.repo_id
118+ ${where.length ? `WHERE ${where.join(" AND ")}` : ""} ORDER BY p.updated_at DESC LIMIT ?`;
119+ params.push(filter.limit ?? 500);
120+ return (await this.db.prepare(q).bind(...params).all<PackageRow & { tag_count: number; version_count: number }>()).results;
121+ }
122+
123+ async updatePackage(id: string, patch: { visibility?: PackageRow["visibility"]; repoId?: string | null; description?: string | null }): Promise<void> {
124+ const sets: string[] = ["updated_at = ?"];
125+ const params: unknown[] = [now()];
126+ if (patch.visibility) {
127+ sets.push("visibility = ?");
128+ params.push(patch.visibility);
129+ }
130+ if (patch.repoId !== undefined) {
131+ sets.push("repo_id = ?");
132+ params.push(patch.repoId);
133+ }
134+ if (patch.description !== undefined) {
135+ sets.push("description = ?");
136+ params.push(patch.description);
137+ }
138+ await this.db.prepare(`UPDATE packages SET ${sets.join(", ")} WHERE id = ?`).bind(...params, id).run();
139+ }
140+
141+ async touchPackage(id: string): Promise<void> {
142+ await this.db.prepare("UPDATE packages SET updated_at = ? WHERE id = ?").bind(now(), id).run();
143+ }
144+
145+ /** Deletes a package and everything it indexes; unreferenced bytes go at the next cleanup. */
146+ async deletePackage(id: string): Promise<void> {
147+ await this.db.batch([
148+ this.db.prepare("DELETE FROM package_tags WHERE package_id = ?").bind(id),
149+ this.db.prepare("DELETE FROM package_manifests WHERE package_id = ?").bind(id),
150+ this.db.prepare("DELETE FROM package_manifest_blobs WHERE package_id = ?").bind(id),
151+ this.db.prepare("DELETE FROM package_blobs WHERE package_id = ?").bind(id),
152+ this.db.prepare("DELETE FROM registry_uploads WHERE package_id = ?").bind(id),
153+ this.db.prepare("DELETE FROM packages WHERE id = ?").bind(id),
154+ ]);
155+ }
156+
157+ // ---------------------------------------------------------------- blobs
158+
159+ async blob(digest: string): Promise<BlobRow | null> {
160+ return this.db.prepare("SELECT * FROM registry_blobs WHERE digest = ?").bind(digest).first<BlobRow>();
161+ }
162+
163+ /** The blob, if this package holds it. */
164+ async packageBlob(packageId: string, digest: string): Promise<BlobRow | null> {
165+ return this.db.prepare("SELECT b.* FROM package_blobs pb JOIN registry_blobs b ON b.digest = pb.digest WHERE pb.package_id = ? AND pb.digest = ?").bind(packageId, digest).first<BlobRow>();
166+ }
167+
168+ async packageBlobs(packageId: string, digests: string[]): Promise<Set<string>> {
169+ if (!digests.length) return new Set();
170+ const found = new Set<string>();
171+ for (let i = 0; i < digests.length; i += 90) {
172+ const chunk = digests.slice(i, i + 90);
173+ const r = await this.db.prepare(`SELECT digest FROM package_blobs WHERE package_id = ? AND digest IN (${chunk.map(() => "?").join(",")})`).bind(packageId, ...chunk).all<{ digest: string }>();
174+ for (const x of r.results) found.add(x.digest);
175+ }
176+ return found;
177+ }
178+
179+ /** Records a verified blob (keeping an existing copy if there is one) and returns the stored row. */
180+ async addBlob(digest: string, size: number, r2Key: string): Promise<BlobRow> {
181+ await this.db.prepare("INSERT INTO registry_blobs (digest, size, r2_key, created_at) VALUES (?, ?, ?, ?) ON CONFLICT(digest) DO NOTHING").bind(digest, size, r2Key, now()).run();
182+ return (await this.blob(digest))!;
183+ }
184+
185+ async linkBlob(packageId: string, digest: string): Promise<void> {
186+ await this.db.prepare("INSERT INTO package_blobs (package_id, digest, created_at) VALUES (?, ?, ?) ON CONFLICT DO NOTHING").bind(packageId, digest, now()).run();
187+ }
188+
189+ async unlinkBlob(packageId: string, digest: string): Promise<void> {
190+ await this.db.prepare("DELETE FROM package_blobs WHERE package_id = ? AND digest = ?").bind(packageId, digest).run();
191+ }
192+
193+ // ---------------------------------------------------------------- uploads
194+
195+ async createUpload(packageId: string): Promise<UploadRow> {
196+ const id = newId();
197+ const t = now();
198+ const row: UploadRow = { id, package_id: packageId, r2_key: blobKey(id), multipart_id: null, parts: "[]", size: 0, tail: 0, digest: null, created_at: t, updated_at: t };
199+ await this.db.prepare("INSERT INTO registry_uploads (id, package_id, r2_key, parts, size, tail, created_at, updated_at) VALUES (?, ?, ?, '[]', 0, 0, ?, ?)").bind(id, packageId, row.r2_key, t, t).run();
200+ return row;
201+ }
202+
203+ async upload(id: string): Promise<UploadRow | null> {
204+ return this.db.prepare("SELECT * FROM registry_uploads WHERE id = ?").bind(id).first<UploadRow>();
205+ }
206+
207+ async saveUpload(u: UploadRow): Promise<void> {
208+ await this.db
209+ .prepare("UPDATE registry_uploads SET multipart_id = ?, parts = ?, size = ?, tail = ?, digest = ?, updated_at = ? WHERE id = ?")
210+ .bind(u.multipart_id, u.parts, u.size, u.tail, u.digest, now(), u.id)
211+ .run();
212+ }
213+
214+ async deleteUpload(id: string): Promise<void> {
215+ await this.db.prepare("DELETE FROM registry_uploads WHERE id = ?").bind(id).run();
216+ }
217+
218+ // ---------------------------------------------------------------- manifests and tags
219+
220+ async manifest(packageId: string, digest: string): Promise<ManifestRow | null> {
221+ return this.db.prepare("SELECT * FROM package_manifests WHERE package_id = ? AND digest = ?").bind(packageId, digest).first<ManifestRow>();
222+ }
223+
224+ async tagDigest(packageId: string, tag: string): Promise<string | null> {
225+ return (await this.db.prepare("SELECT digest FROM package_tags WHERE package_id = ? AND tag = ?").bind(packageId, tag).first<{ digest: string }>())?.digest ?? null;
226+ }
227+
228+ /** Which of these manifests the package holds, with the size of each one's content. */
229+ async manifestSizes(packageId: string, digests: string[]): Promise<Map<string, number>> {
230+ const found = new Map<string, number>();
231+ for (let i = 0; i < digests.length; i += 90) {
232+ const chunk = digests.slice(i, i + 90);
233+ const r = await this.db.prepare(`SELECT digest, total_size FROM package_manifests WHERE package_id = ? AND digest IN (${chunk.map(() => "?").join(",")})`).bind(packageId, ...chunk).all<{ digest: string; total_size: number }>();
234+ for (const x of r.results) found.set(x.digest, x.total_size);
235+ }
236+ return found;
237+ }
238+
239+ async putManifest(m: ManifestRow, tag: string | null, blobs: string[]): Promise<void> {
240+ const t = now();
241+ const stmts = [
242+ ...blobs.map((d) => this.db.prepare("INSERT INTO package_manifest_blobs (package_id, manifest, digest) VALUES (?, ?, ?) ON CONFLICT DO NOTHING").bind(m.package_id, m.digest, d)),
243+ this.db
244+ .prepare("INSERT INTO package_manifests (package_id, digest, media_type, artifact_type, subject, annotations, size, total_size, platforms, pushed_by, created_at) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) ON CONFLICT(package_id, digest) DO NOTHING")
245+ .bind(m.package_id, m.digest, m.media_type, m.artifact_type, m.subject, m.annotations, m.size, m.total_size, m.platforms, m.pushed_by, m.created_at),
246+ this.db.prepare("UPDATE packages SET updated_at = ? WHERE id = ?").bind(t, m.package_id),
247+ ];
248+ if (tag) stmts.push(this.db.prepare("INSERT INTO package_tags (package_id, tag, digest, updated_at) VALUES (?, ?, ?, ?) ON CONFLICT(package_id, tag) DO UPDATE SET digest = excluded.digest, updated_at = excluded.updated_at").bind(m.package_id, tag, m.digest, t));
249+ await this.db.batch(stmts);
250+ }
251+
252+ async tags(packageId: string, opts: { n?: number; last?: string } = {}): Promise<string[]> {
253+ const r = await this.db
254+ .prepare("SELECT tag FROM package_tags WHERE package_id = ? AND tag > ? ORDER BY tag LIMIT ?")
255+ .bind(packageId, opts.last ?? "", Math.min(Math.max(opts.n ?? 10000, 0), 10000))
256+ .all<{ tag: string }>();
257+ return r.results.map((x) => x.tag);
258+ }
259+
260+ async deleteTag(packageId: string, tag: string): Promise<boolean> {
261+ const r = await this.db.prepare("DELETE FROM package_tags WHERE package_id = ? AND tag = ?").bind(packageId, tag).run();
262+ return (r.meta.changes ?? 0) > 0;
263+ }
264+
265+ /** Deletes a manifest and the tags pointing at it. */
266+ async deleteManifest(packageId: string, digest: string): Promise<boolean> {
267+ const [, r] = await this.db.batch([
268+ this.db.prepare("DELETE FROM package_tags WHERE package_id = ? AND digest = ?").bind(packageId, digest),
269+ this.db.prepare("DELETE FROM package_manifests WHERE package_id = ? AND digest = ?").bind(packageId, digest),
270+ this.db.prepare("DELETE FROM package_manifest_blobs WHERE package_id = ? AND manifest = ?").bind(packageId, digest),
271+ ]);
272+ return (r!.meta.changes ?? 0) > 0;
273+ }
274+
275+ async versionById(packageId: string, id: number): Promise<(ManifestRow & { tags: string[] }) | null> {
276+ const m = await this.db.prepare("SELECT rowid AS id, * FROM package_manifests WHERE package_id = ? AND rowid = ?").bind(packageId, id).first<ManifestRow>();
277+ if (!m) return null;
278+ const tags = (await this.db.prepare("SELECT tag FROM package_tags WHERE package_id = ? AND digest = ? ORDER BY tag").bind(packageId, m.digest).all<{ tag: string }>()).results.map((t) => t.tag);
279+ return { ...m, tags };
280+ }
281+
282+ async referrers(packageId: string, subject: string): Promise<ManifestRow[]> {
283+ return (await this.db.prepare("SELECT * FROM package_manifests WHERE package_id = ? AND subject = ? ORDER BY created_at").bind(packageId, subject).all<ManifestRow>()).results;
284+ }
285+
286+ /** Versions (manifests) with their tags, newest first. Children of multi-platform images are left out. */
287+ async versions(packageId: string, opts: { limit?: number; all?: boolean } = {}): Promise<Array<ManifestRow & { tags: string[] }>> {
288+ const manifests = (await this.db.prepare("SELECT rowid AS id, * FROM package_manifests WHERE package_id = ? ORDER BY created_at DESC LIMIT ?").bind(packageId, opts.limit ?? 1000).all<ManifestRow>()).results;
289+ const tags = (await this.db.prepare("SELECT tag, digest FROM package_tags WHERE package_id = ? ORDER BY updated_at DESC").bind(packageId).all<{ tag: string; digest: string }>()).results;
290+ const byDigest = new Map<string, string[]>();
291+ for (const t of tags) byDigest.set(t.digest, [...(byDigest.get(t.digest) ?? []), t.tag]);
292+ const children = new Set<string>();
293+ if (!opts.all) {
294+ for (const m of manifests) {
295+ for (const d of (m.platforms ? (JSON.parse(m.platforms) as Array<{ digest?: string }>) : []).map((p) => p.digest)) if (d) children.add(d);
296+ if (m.subject) children.add(m.digest);
297+ }
298+ }
299+ return manifests.filter((m) => byDigest.has(m.digest) || !children.has(m.digest)).map((m) => ({ ...m, tags: byDigest.get(m.digest) ?? [] }));
300+ }
301+
302+ // ---------------------------------------------------------------- cleanup
303+
304+ /**
305+ * Removes abandoned uploads and bytes nothing references any more. Runs from the
306+ * cron trigger; a grace period keeps blobs that a push in progress is about to use.
307+ */
308+ async cleanup(limit = 200): Promise<{ uploads: number; blobs: number; manifests: number }> {
309+ const day = new Date(Date.now() - 86400_000).toISOString();
310+ const stale = (await this.db.prepare("SELECT * FROM registry_uploads WHERE updated_at < ? LIMIT ?").bind(day, limit).all<UploadRow>()).results;
311+ for (const u of stale) {
312+ if (u.multipart_id) await this.env.BUCKET.resumeMultipartUpload(u.r2_key, u.multipart_id).abort().catch(() => {});
313+ await this.env.BUCKET.delete([u.r2_key, tailKey(u.id)]);
314+ await this.deleteUpload(u.id);
315+ }
316+ // Blobs a package holds that none of its manifests use any more (deleted versions, abandoned pushes).
317+ await this.db
318+ .prepare("DELETE FROM package_blobs WHERE rowid IN (SELECT pb.rowid FROM package_blobs pb WHERE pb.created_at < ? AND NOT EXISTS (SELECT 1 FROM package_manifest_blobs mb WHERE mb.package_id = pb.package_id AND mb.digest = pb.digest) LIMIT ?)")
319+ .bind(day, limit * 5)
320+ .run();
321+ // Blobs no package holds, past the grace period.
322+ const blobs = (await this.db.prepare("SELECT b.* FROM registry_blobs b WHERE b.created_at < ? AND NOT EXISTS (SELECT 1 FROM package_blobs pb WHERE pb.digest = b.digest) LIMIT ?").bind(day, limit).all<BlobRow>()).results;
323+ for (const b of blobs) {
324+ await this.db.prepare("DELETE FROM registry_blobs WHERE digest = ? AND NOT EXISTS (SELECT 1 FROM package_blobs pb WHERE pb.digest = ?)").bind(b.digest, b.digest).run();
325+ if (!(await this.blob(b.digest))) await this.env.BUCKET.delete(b.r2_key);
326+ }
327+ // Manifest bytes no package lists. They are tiny, so only the R2 listing costs anything; do a page per run.
328+ let manifests = 0;
329+ // A random shard each run, so every manifest gets looked at over time.
330+ const shard = [...crypto.getRandomValues(new Uint8Array(1))].map((b) => b.toString(16).padStart(2, "0")).join("");
331+ const listed = await this.env.BUCKET.list({ prefix: `registry/manifests/sha256/${shard}`, limit: 500 });
332+ const old = listed.objects.filter((o) => o.uploaded.toISOString() < day);
333+ if (old.length) {
334+ const digests = old.map((o) => o.key.slice("registry/manifests/".length).replace("/", ":"));
335+ const inUse = new Set<string>();
336+ for (let i = 0; i < digests.length; i += 90) {
337+ const chunk = digests.slice(i, i + 90);
338+ const r = await this.db.prepare(`SELECT DISTINCT digest FROM package_manifests WHERE digest IN (${chunk.map(() => "?").join(",")})`).bind(...chunk).all<{ digest: string }>();
339+ for (const x of r.results) inUse.add(x.digest);
340+ }
341+ const unused = old.filter((_, i) => !inUse.has(digests[i]!)).map((o) => o.key);
342+ if (unused.length) await this.env.BUCKET.delete(unused);
343+ manifests = unused.length;
344+ }
345+ return { uploads: stale.length, blobs: blobs.length, manifests };
346+ }
347+}
addedapps/diffinery/src/registry/upload.ts+104−0View file
@@ -1,0 +1,104 @@
1+import type { Env } from "../env.js";
2+import { tailKey, type UploadRow } from "./store.js";
3+
4+/**
5+ * Blob uploads stream into R2. Small blobs become one object; bigger ones become a
6+ * multipart upload with fixed-size parts (R2 wants every part but the last to be the
7+ * same size). Bytes that don't fill a part yet wait in a tail object until the next
8+ * chunk or the final PUT. The object key is the upload's, not the digest's: R2 can't
9+ * rename, so the blob index in D1 records where each digest's bytes are.
10+ */
11+export const PART_SIZE = 10 * 1024 * 1024;
12+
13+export class UploadError extends Error {}
14+
15+const hex = (b: ArrayBuffer) => [...new Uint8Array(b)].map((x) => x.toString(16).padStart(2, "0")).join("");
16+
17+/**
18+ * Appends a request body to an upload. When the upload started with this request the
19+ * SHA-256 is computed on the way through; chunked uploads are hashed when they finish.
20+ */
21+export async function ingest(env: Env, u: UploadRow, body: ReadableStream<Uint8Array> | null, final: boolean): Promise<UploadRow> {
22+ const bucket = env.BUCKET;
23+ const parts = JSON.parse(u.parts) as R2UploadedPart[];
24+ let mp = u.multipart_id ? bucket.resumeMultipartUpload(u.r2_key, u.multipart_id) : null;
25+ let buf = new Uint8Array(PART_SIZE);
26+ let fill = 0;
27+
28+ // Bring back the bytes that are not in a finished part yet.
29+ const pending = mp ? u.tail : u.size;
30+ if (pending > 0) {
31+ const obj = await bucket.get(mp ? tailKey(u.id) : u.r2_key);
32+ if (!obj) throw new UploadError("the upload's data is missing; start the upload again");
33+ const bytes = new Uint8Array(await obj.arrayBuffer());
34+ buf.set(bytes);
35+ fill = bytes.length;
36+ }
37+
38+ const hasher = u.size === 0 ? new crypto.DigestStream("SHA-256") : null;
39+ const hw = hasher?.getWriter();
40+ let added = 0;
41+ const flush = async () => {
42+ mp ??= await bucket.createMultipartUpload(u.r2_key, { httpMetadata: { contentType: "application/octet-stream" } });
43+ parts.push(await mp.uploadPart(parts.length + 1, buf));
44+ buf = new Uint8Array(PART_SIZE);
45+ fill = 0;
46+ };
47+
48+ if (body) {
49+ const reader = body.getReader();
50+ for (;;) {
51+ const { done, value } = await reader.read();
52+ if (done) break;
53+ if (!value?.length) continue;
54+ added += value.length;
55+ if (hw) await hw.write(value);
56+ for (let off = 0; off < value.length; ) {
57+ const n = Math.min(PART_SIZE - fill, value.length - off);
58+ buf.set(value.subarray(off, off + n), fill);
59+ fill += n;
60+ off += n;
61+ if (fill === PART_SIZE) await flush();
62+ }
63+ }
64+ }
65+ let digest = u.digest;
66+ if (hw && hasher) {
67+ await hw.close();
68+ digest = `sha256:${hex(await hasher.digest)}`;
69+ } else if (added > 0) digest = null;
70+
71+ const hadTail = !!u.multipart_id && u.tail > 0;
72+ if (final) {
73+ if (mp) {
74+ if (fill > 0 || !parts.length) parts.push(await mp.uploadPart(parts.length + 1, buf.subarray(0, fill)));
75+ await mp.complete(parts);
76+ if (hadTail) await bucket.delete(tailKey(u.id));
77+ } else if (added > 0 || u.size === 0) {
78+ await bucket.put(u.r2_key, buf.subarray(0, fill), { httpMetadata: { contentType: "application/octet-stream" } });
79+ }
80+ } else if (mp) {
81+ if (fill > 0) await bucket.put(tailKey(u.id), buf.subarray(0, fill));
82+ else if (hadTail) await bucket.delete(tailKey(u.id));
83+ } else if (added > 0) {
84+ await bucket.put(u.r2_key, buf.subarray(0, fill));
85+ }
86+ // A small upload that grew into a multipart one: its first bytes went into part 1, so the
87+ // single object written by an earlier chunk is replaced when the multipart upload completes.
88+ return { ...u, multipart_id: mp?.uploadId ?? null, parts: JSON.stringify(parts), size: u.size + added, tail: mp && !final ? fill : 0, digest };
89+}
90+
91+/** SHA-256 of a stored object, for uploads that arrived in several requests. */
92+export async function digestOf(env: Env, key: string): Promise<{ digest: string; size: number } | null> {
93+ const obj = await env.BUCKET.get(key);
94+ if (!obj) return null;
95+ const ds = new crypto.DigestStream("SHA-256");
96+ await obj.body.pipeTo(ds);
97+ return { digest: `sha256:${hex(await ds.digest)}`, size: obj.size };
98+}
99+
100+/** Throws away an upload's bytes. */
101+export async function discard(env: Env, u: UploadRow): Promise<void> {
102+ if (u.multipart_id) await env.BUCKET.resumeMultipartUpload(u.r2_key, u.multipart_id).abort().catch(() => {});
103+ await env.BUCKET.delete([u.r2_key, tailKey(u.id)]);
104+}
modifiedapps/diffinery/src/repo/RepoDO.ts+3−0View file
@@ -692,6 +692,9 @@
692692 runnerComplete(...a: Parameters<Actions["runnerComplete"]>) {
693693 return this.actions.runnerComplete(...a);
694694 }
695+ packagesPermission(...a: Parameters<Actions["packagesPermission"]>) {
696+ return this.actions.packagesPermission(...a);
697+ }
695698 runnerAuth(...a: Parameters<Actions["runnerAuth"]>) {
696699 return this.actions.runnerAuth(...a);
697700 }
modifiedapps/diffinery/src/repo/actions.ts+17−0View file
@@ -1028,6 +1028,23 @@
10281028 return { allowed, github };
10291029 }
10301030
1031+ /**
1032+ * What a job token may do with container packages (the `packages` permission).
1033+ * Like GitHub's permissive default, a workflow without a permissions block gets write;
1034+ * a permissions map that leaves `packages` out gets none.
1035+ */
1036+ async packagesPermission(token: string): Promise<"none" | "read" | "write"> {
1037+ const auth = await this.runnerAuth(token);
1038+ if (!auth || auth.kind !== "job") return "none";
1039+ const job = this.jobRow(auth.jobId);
1040+ const exp = await this.expanded(this.runRow(job.run_id));
1041+ const perms = exp.workflow.jobs[job.job_key]?.permissions ?? exp.workflowOf.get(job.job_key)?.permissions;
1042+ if (perms === undefined || perms === "write-all") return "write";
1043+ if (perms === "read-all") return "read";
1044+ const v = perms.packages;
1045+ return v === "write" || v === "read" ? v : "none";
1046+ }
1047+
10311048 async runnerSpec(jobId: string, token: string): Promise<JobSpec> {
10321049 const job = await this.authJob(jobId, token);
10331050 const run = this.runRow(job.run_id);
modifiedapps/diffinery/src/web/account.tsx+10−4View file
@@ -1406,11 +1406,16 @@
14061406 <div>
14071407 <div class="row between mb">
14081408 <h2 class="m0">Repositories</h2>
1409- {acct.id === me.id ? (
1410- <a class="btn sm primary" href="/new">
1411- {raw(ICONS.plus)} New
1409+ <div class="row">
1410+ <a class="btn sm" href={`/${acct.login}/-/packages`}>
1411+ {raw(ICONS.package)} Packages
14121412 </a>
1413- ) : null}
1413+ {acct.id === me.id ? (
1414+ <a class="btn sm primary" href="/new">
1415+ {raw(ICONS.plus)} New
1416+ </a>
1417+ ) : null}
1418+ </div>
14141419 </div>
14151420 <div class="box">
14161421 <RepoList repos={repos} showOwner={false} />
@@ -1674,6 +1679,7 @@
16741679 {label}
16751680 </a>
16761681 ))}
1682+ <a href={`${base}/-/packages`}>Packages</a>
16771683 </nav>
16781684 {body}
16791685 </>,
modifiedapps/diffinery/src/web/assets.ts+1−0View file
@@ -378,6 +378,7 @@
378378 export const ICONS = {
379379 logo: `<svg viewBox="0 0 24 24" width="22" height="22" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" aria-hidden="true"><circle cx="6" cy="6" r="2.5"/><circle cx="6" cy="18" r="2.5"/><circle cx="18" cy="12" r="2.5"/><path d="M6 8.5v7M8.3 7.2c4 1 7.4 2.2 7.6 3.3M8.3 16.8c4-1 7.4-2.2 7.6-3.3"/></svg>`,
380380 success: I('<circle cx="8" cy="8" r="6.5"/><path d="M5.2 8.2l1.9 1.9 3.7-4"/>'),
381+ package: I('<path d="M8 1.5l6 3v7l-6 3-6-3v-7z"/><path d="M2 4.5l6 3 6-3M8 7.5v7"/>'),
381382 failure: I('<circle cx="8" cy="8" r="6.5"/><path d="M5.8 5.8l4.4 4.4M10.2 5.8l-4.4 4.4"/>'),
382383 running: I('<circle cx="8" cy="8" r="6.5" opacity=".25"/><path d="M8 1.5a6.5 6.5 0 0 1 6.5 6.5"/>'),
383384 queued: I('<circle cx="8" cy="8" r="6.5"/><circle cx="8" cy="8" r="1.6" fill="currentColor" stroke="none"/>'),
addedapps/diffinery/src/web/packages.tsx+358−0View file
@@ -1,0 +1,358 @@
1+/** @jsxImportSource hono/jsx */
2+import { raw } from "hono/html";
3+import type { Child } from "hono/jsx";
4+import type { AppContext } from "../auth/principal.js";
5+import type { Account } from "../lib/directory.js";
6+import { loadRepo } from "../http/common.js";
7+import { HttpError } from "../lib/util.js";
8+import { allows, packageAccess, type PackageRole } from "../registry/access.js";
9+import { RegistryStore, type ManifestRow, type PackageRow } from "../registry/store.js";
10+import { ICONS } from "./assets.js";
11+import { KSelect } from "./kumo.js";
12+import { csrfToken } from "./csrf.js";
13+import { Avatar, CopyButton, Form, Time, act, fmtBytes, post, principalAccount, prm, render, repoUrl, type WebApp } from "./ui.js";
14+
15+/**
16+ * Container packages in the web UI: an owner's packages, one package with its tags and
17+ * versions (and settings for admins), and the packages linked to a repository.
18+ */
19+
20+const host = (c: AppContext) => new URL(c.env.PUBLIC_URL).host;
21+const imageRef = (c: AppContext, p: Pick<PackageRow, "owner_login" | "name">) => `${host(c)}/${p.owner_login.toLowerCase()}/${p.name}`;
22+const pkgUrl = (p: Pick<PackageRow, "owner_login" | "name">) => `/${p.owner_login}/-/packages/container/${p.name}`;
23+
24+async function ownerOf(c: AppContext): Promise<Account> {
25+ const owner = await c.get("dir").accountByLogin(prm(c, "owner"));
26+ if (!owner || owner.kind === "agent") throw new HttpError(404, "not found");
27+ return owner;
28+}
29+
30+async function visible(c: AppContext, owner: Account, rows: PackageRow[]): Promise<PackageRow[]> {
31+ const out: PackageRow[] = [];
32+ for (const p of rows) if (allows((await packageAccess(c, owner, p.name, p)).role, "pull")) out.push(p);
33+ return out;
34+}
35+
36+async function loadPackage(c: AppContext, owner: Account, name: string, need: PackageRole = "pull") {
37+ const store = new RegistryStore(c.env);
38+ const pkg = await store.package(owner.id, name.toLowerCase());
39+ if (!pkg) throw new HttpError(404, "package not found");
40+ const access = await packageAccess(c, owner, pkg.name, pkg);
41+ if (!allows(access.role, "pull")) throw new HttpError(c.get("principal").type === "anonymous" ? 401 : 404, "package not found");
42+ if (!allows(access.role, need)) throw new HttpError(403, "package admins only");
43+ return { store, pkg, access };
44+}
45+
46+const VisIcon = ({ v }: { v: string }) => <span class="st faint">{raw(v === "public" ? ICONS.globe : v === "internal" ? ICONS.org : ICONS.lock)}</span>;
47+
48+const PackageList = ({ c, rows, showRepo, empty }: { c: AppContext; rows: Array<PackageRow & { version_count?: number; tag_count?: number }>; showRepo: boolean; empty: Child }) =>
49+ rows.length ? (
50+ <ul class="list">
51+ {rows.map((p) => (
52+ <li>
53+ <span class="st faint">{raw(ICONS.package)}</span>
54+ <div class="grow">
55+ <a href={pkgUrl(p)}>
56+ <b>
57+ {p.owner_login}/{p.name}
58+ </b>
59+ </a>{" "}
60+ <span class="badge">{p.visibility}</span>
61+ {p.description ? <div class="muted small">{p.description}</div> : null}
62+ <div class="muted small mono">{imageRef(c, p)}</div>
63+ </div>
64+ {showRepo && p.repo_name ? (
65+ <a class="small muted" href={`/${p.owner_login}/${p.repo_name}`}>
66+ {raw(ICONS.repo)} {p.repo_name}
67+ </a>
68+ ) : null}
69+ <span class="muted small">
70+ {p.tag_count ?? 0} tag{p.tag_count === 1 ? "" : "s"} · updated <Time at={p.updated_at} />
71+ </span>
72+ </li>
73+ ))}
74+ </ul>
75+ ) : (
76+ <div class="empty">{empty}</div>
77+ );
78+
79+const Usage = ({ c, image, tag }: { c: AppContext; image: string; tag: string }) => {
80+ const login = `docker login ${host(c)} -u <username> -p <token>`;
81+ const pull = `docker pull ${image}:${tag}`;
82+ const workflow = `- name: Sign in to the Diffinery registry
83+ run: echo "\${{ secrets.GITHUB_TOKEN }}" | docker login ${host(c)} -u "\${{ github.actor }}" --password-stdin
84+- run: |
85+ docker build -t ${image}:\${{ github.sha }} .
86+ docker push ${image}:\${{ github.sha }}`;
87+ return (
88+ <div class="box">
89+ <div class="head">
90+ <h3>Use this image</h3>
91+ </div>
92+ <div class="body">
93+ <p class="muted small m0">Sign in with any username and a token from Settings → Tokens as the password (the read scope pulls, write pushes).</p>
94+ <div class="row mt">
95+ <pre class="secret grow m0 mono">{login}</pre>
96+ </div>
97+ <div class="row mt">
98+ <pre class="secret grow m0 mono">{pull}</pre>
99+ <CopyButton text={pull} />
100+ </div>
101+ <p class="muted small mt">
102+ In a workflow, the job's <code>GITHUB_TOKEN</code> signs in. It can push to packages of its own repository (with <code>permissions: packages: write</code> when the workflow sets permissions) and pull its owner's packages.
103+ </p>
104+ <pre class="secret m0 mono small">{workflow}</pre>
105+ </div>
106+ </div>
107+ );
108+};
109+
110+const platformsOf = (v: ManifestRow) => (v.platforms ? (JSON.parse(v.platforms) as Array<{ platform: string | null }>).map((x) => x.platform).filter((x): x is string => !!x && x !== "unknown/unknown") : []);
111+
112+export function registerPackages(web: WebApp): void {
113+ web.get("/:owner/-/packages", async (c) => {
114+ const owner = await ownerOf(c);
115+ const rows = await visible(c, owner, (await new RegistryStore(c.env).packages({ ownerIds: [owner.id], limit: 1000 })).filter((p) => p.owner_id === owner.id));
116+ return render(
117+ c,
118+ { title: `${owner.login} packages` },
119+ <>
120+ <div class="row mb" style="gap:12px">
121+ <Avatar login={owner.login} size="lg" />
122+ <div>
123+ <h1 class="m0">
124+ <a href={`/${owner.login}`}>{owner.login}</a> <span class="faint">/</span> Packages
125+ </h1>
126+ <div class="muted small">
127+ Container images at <span class="mono">{host(c)}/{owner.login.toLowerCase()}/…</span>
128+ </div>
129+ </div>
130+ </div>
131+ <div class="box">
132+ <PackageList
133+ c={c}
134+ rows={rows}
135+ showRepo
136+ empty={
137+ <>
138+ No packages yet. Push one with <code class="mono">docker push {host(c)}/{owner.login.toLowerCase()}/IMAGE:TAG</code>.
139+ </>
140+ }
141+ />
142+ </div>
143+ </>,
144+ );
145+ });
146+
147+ web.get("/:owner/-/packages/container/:name{.+}", async (c) => {
148+ const owner = await ownerOf(c);
149+ const { store, pkg, access } = await loadPackage(c, owner, prm(c, "name"));
150+ const versions = await store.versions(pkg.id);
151+ const tagged = versions.flatMap((v) => v.tags.map((t) => ({ tag: t, v })));
152+ const isAdmin = allows(access.role, "admin");
153+ const image = imageRef(c, pkg);
154+ const t = csrfToken(c);
155+ const me = principalAccount(c);
156+ const repos = isAdmin && me ? await c.get("dir").reposVisibleTo(me, owner.id) : [];
157+ const latest = tagged.find((x) => x.tag === "latest")?.tag ?? tagged[0]?.tag ?? "latest";
158+ return render(
159+ c,
160+ { title: `${owner.login}/${pkg.name}`, description: pkg.description ?? `Container image ${image}` },
161+ <>
162+ <div class="mb">
163+ <h1 class="m0">
164+ <VisIcon v={pkg.visibility} />
165+ <a href={`/${owner.login}/-/packages`}>{owner.login}</a> <span class="faint">/</span> <strong>{pkg.name}</strong> <span class="badge">{pkg.visibility}</span>
166+ </h1>
167+ {pkg.description ? <p class="muted m0">{pkg.description}</p> : null}
168+ <p class="muted small">
169+ {pkg.repo_name ? (
170+ <>
171+ Linked to{" "}
172+ <a href={`/${owner.login}/${pkg.repo_name}`}>
173+ {owner.login}/{pkg.repo_name}
174+ </a>
175+ , which sets who can pull and push.{" "}
176+ </>
177+ ) : (
178+ <>Not linked to a repository. </>
179+ )}
180+ Created <Time at={pkg.created_at} />, updated <Time at={pkg.updated_at} />.
181+ </p>
182+ </div>
183+ <Usage c={c} image={image} tag={latest} />
184+ <div class="box">
185+ <div class="head">
186+ <h3>
187+ {raw(ICONS.tag)} Versions <span class="muted small">({versions.length})</span>
188+ </h3>
189+ </div>
190+ {versions.length ? (
191+ <table class="tbl">
192+ <thead>
193+ <tr>
194+ <th>Tags</th>
195+ <th>Digest</th>
196+ <th>Platforms</th>
197+ <th>Size</th>
198+ <th>Pushed</th>
199+ {isAdmin ? <th /> : null}
200+ </tr>
201+ </thead>
202+ <tbody>
203+ {versions.map((v) => (
204+ <tr>
205+ <td>{v.tags.length ? v.tags.map((x) => <span class="badge green">{x}</span>) : <span class="muted small">untagged</span>}</td>
206+ <td class="mono small">
207+ <span title={v.digest}>{v.digest.slice(0, 19)}</span> <CopyButton text={`${image}@${v.digest}`} label="Copy" />
208+ </td>
209+ <td class="small">{platformsOf(v).join(", ") || "—"}</td>
210+ <td class="small">{fmtBytes(v.total_size)}</td>
211+ <td class="small">
212+ <Time at={v.created_at} />
213+ {v.pushed_by ? <span class="muted"> by {v.pushed_by}</span> : null}
214+ </td>
215+ {isAdmin ? (
216+ <td class="right">
217+ <Form action={`/${owner.login}/-/packages/-/delete-version`} t={t} class="inline" confirm={`Delete ${v.tags.length ? v.tags.join(", ") : v.digest.slice(0, 19)}? Anything pulling it by tag or digest will stop working.`}>
218+ <input type="hidden" name="name" value={pkg.name} />
219+ <input type="hidden" name="digest" value={v.digest} />
220+ <button class="btn sm danger">Delete</button>
221+ </Form>
222+ </td>
223+ ) : null}
224+ </tr>
225+ ))}
226+ </tbody>
227+ </table>
228+ ) : (
229+ <div class="empty">No versions yet.</div>
230+ )}
231+ </div>
232+ {isAdmin ? (
233+ <>
234+ <Form action={`/${owner.login}/-/packages/-/update`} t={t} class="box">
235+ <div class="head">
236+ <h3>{raw(ICONS.gear)} Settings</h3>
237+ </div>
238+ <div class="body">
239+ <input type="hidden" name="name" value={pkg.name} />
240+ <div class="field" style="max-width:360px">
241+ <label>Visibility</label>
242+ <KSelect
243+ name="visibility"
244+ block
245+ label="Visibility"
246+ value={pkg.visibility}
247+ options={[
248+ { value: "private", label: "Private: people with access to the linked repository or the owner" },
249+ { value: "internal", label: "Internal: anyone signed in to Diffinery" },
250+ { value: "public", label: "Public: anyone, without signing in" },
251+ ]}
252+ />
253+ </div>
254+ <div class="field" style="max-width:360px">
255+ <label>Linked repository</label>
256+ <KSelect name="repository" block label="Linked repository" value={pkg.repo_name ?? ""} options={[{ value: "", label: "None" }, ...repos.map((r) => ({ value: r.name, label: `${r.owner_login}/${r.name}` }))]} />
257+ <div class="hint">The package shares the repository's permissions: readers pull, writers and its workflows push, admins manage.</div>
258+ </div>
259+ <button class="btn primary">Save</button>
260+ </div>
261+ </Form>
262+ <Form action={`/${owner.login}/-/packages/-/delete`} t={t} class="box" confirm={`Delete ${owner.login}/${pkg.name} and all of its versions?`} match={pkg.name}>
263+ <div class="head">
264+ <h3>Delete this package</h3>
265+ </div>
266+ <div class="body row">
267+ <input type="hidden" name="name" value={pkg.name} />
268+ <p class="muted small grow m0">Removes every tag and version. Type the package name to confirm.</p>
269+ <button class="btn danger">Delete package</button>
270+ </div>
271+ </Form>
272+ </>
273+ ) : null}
274+ </>,
275+ );
276+ });
277+
278+ post(web, "/:owner/-/packages/-/update", async (c, f) => {
279+ const owner = await ownerOf(c);
280+ const name = f.s("name");
281+ return act(
282+ c,
283+ pkgUrl({ owner_login: owner.login, name }),
284+ async () => {
285+ const { store, pkg, access } = await loadPackage(c, owner, name, "admin");
286+ const visibility = (["private", "internal", "public"] as const).find((v) => v === f.s("visibility"));
287+ const repoName = f.s("repository").trim();
288+ const repoId = repoName ? (await loadRepo(c, owner.login, repoName, "admin")).repo.id : null;
289+ await store.updatePackage(pkg.id, { visibility, repoId });
290+ await c.get("dir").audit({ actorId: access.actor?.id, actorLogin: access.actor?.login, action: "package.update", target: `${owner.login}/${pkg.name}`, detail: { visibility, repository: repoName || null } });
291+ },
292+ "Package settings saved.",
293+ );
294+ });
295+
296+ post(web, "/:owner/-/packages/-/delete-version", async (c, f) => {
297+ const owner = await ownerOf(c);
298+ const name = f.s("name");
299+ return act(
300+ c,
301+ pkgUrl({ owner_login: owner.login, name }),
302+ async () => {
303+ const { store, pkg, access } = await loadPackage(c, owner, name, "admin");
304+ const digest = f.s("digest");
305+ if (!(await store.deleteManifest(pkg.id, digest))) throw new HttpError(404, "version not found");
306+ await c.get("dir").audit({ actorId: access.actor?.id, actorLogin: access.actor?.login, action: "package.delete_version", target: `${owner.login}/${pkg.name}@${digest}` });
307+ },
308+ "Version deleted.",
309+ );
310+ });
311+
312+ post(web, "/:owner/-/packages/-/delete", async (c, f) => {
313+ const owner = await ownerOf(c);
314+ const name = f.s("name");
315+ return act(
316+ c,
317+ pkgUrl({ owner_login: owner.login, name }),
318+ async () => {
319+ const { store, pkg, access } = await loadPackage(c, owner, name, "admin");
320+ await store.deletePackage(pkg.id);
321+ await c.get("dir").audit({ actorId: access.actor?.id, actorLogin: access.actor?.login, action: "package.delete", target: `${owner.login}/${pkg.name}` });
322+ return `/${owner.login}/-/packages`;
323+ },
324+ "Package deleted.",
325+ );
326+ });
327+
328+ web.get("/:owner/:repo/packages", async (c) => {
329+ const { repo, caller } = await loadRepo(c, prm(c, "owner"), prm(c, "repo"), "read");
330+ const rows = await new RegistryStore(c.env).packages({ repoId: repo.id });
331+ const image = `${host(c)}/${repo.owner_login.toLowerCase()}/${repo.name.toLowerCase()}`;
332+ return render(
333+ c,
334+ { title: `Packages · ${repo.owner_login}/${repo.name}`, repo: { repo, caller, tab: "packages" } },
335+ <>
336+ <div class="box">
337+ <div class="head">
338+ <h3>{raw(ICONS.package)} Packages</h3>
339+ </div>
340+ <PackageList
341+ c={c}
342+ rows={rows}
343+ showRepo={false}
344+ empty={
345+ <>
346+ No packages yet. Images pushed as <code class="mono">{image}</code> (or <code class="mono">{image}/…</code>), or by this repository's workflows, show up here.
347+ </>
348+ }
349+ />
350+ </div>
351+ <Usage c={c} image={image} tag="latest" />
352+ <p class="muted small">
353+ <a href={`${repoUrl(repo)}`}>Back to the code</a> · <a href={`/${repo.owner_login}/-/packages`}>All of {repo.owner_login}'s packages</a>
354+ </p>
355+ </>,
356+ );
357+ });
358+}
modifiedapps/diffinery/src/web/routes.tsx+2−0View file
@@ -11,6 +11,7 @@
1111 import { registerSeo } from "./seo.js";
1212 import { errorPage } from "./ui.js";
1313 import { registerKumoAssets } from "./kumo.js";
14+import { registerPackages } from "./packages.js";
1415
1516 /**
1617 * Server-rendered web UI. Mount at "/" after the API, Git and runner routes:
@@ -31,6 +32,7 @@
3132 // registered before the /:owner/:repo patterns they would otherwise match.
3233 registerKumoAssets(web);
3334 registerAuth(web);
35+registerPackages(web);
3436 registerAccount(web);
3537 registerRepoSettings(web);
3638 registerActions(web);
modifiedapps/diffinery/src/web/ui.tsx+4−1View file
@@ -240,7 +240,7 @@
240240 export interface RepoCtx {
241241 repo: Repo;
242242 caller: Caller;
243- tab?: "code" | "issues" | "pulls" | "actions" | "settings" | "";
243+ tab?: "code" | "issues" | "pulls" | "actions" | "packages" | "settings" | "";
244244 /** Current ref, used by the palette's "go to file". */
245245 ref?: string;
246246 counts?: { issues?: number; pulls?: number };
@@ -293,6 +293,9 @@
293293 <a href={`${base}/actions`} class={tab === "actions" ? "on" : ""}>
294294 {raw(ICONS.play)} Actions
295295 </a>
296+ <a href={`${base}/packages`} class={tab === "packages" ? "on" : ""}>
297+ {raw(ICONS.package)} Packages
298+ </a>
296299 {atLeast(r.caller.role, "admin") ? (
297300 <a href={`${base}/settings`} class={tab === "settings" ? "on" : ""}>
298301 {raw(ICONS.gear)} Settings
addedapps/diffinery/test/e2e/registry.test.mjs+484−0View file
@@ -1,0 +1,484 @@
1+// End-to-end container registry: a real `wrangler dev` Worker, raw OCI distribution API
2+// calls, the docker CLI (when Docker is available) and workflows whose GITHUB_TOKEN
3+// pushes and pulls, run by the real runner agent.
4+//
5+// pnpm --filter @diffinery/runner build && pnpm --filter @diffinery/app e2e
6+//
7+// Covers: monolithic, chunked and streamed blob uploads, digest and range checks,
8+// manifests by tag and digest, indexes, referrers, tag listing, cross-repository mounts,
9+// token scopes, the packages REST API, the job token's `packages` permission, linking
10+// packages to repositories, and docker login/push/pull.
11+import { spawn, spawnSync, execFileSync } from "node:child_process";
12+import { createHash, randomBytes } from "node:crypto";
13+import { mkdtempSync, writeFileSync, mkdirSync, rmSync, existsSync, readFileSync } from "node:fs";
14+import { tmpdir } from "node:os";
15+import { join, dirname, resolve } from "node:path";
16+import { fileURLToPath } from "node:url";
17+import { createServer } from "node:net";
18+import { test, before, after } from "node:test";
19+import assert from "node:assert/strict";
20+
21+const APP = resolve(dirname(fileURLToPath(import.meta.url)), "../..");
22+const AGENT = resolve(APP, "../../containers/runner/dist/agent.mjs");
23+const work = mkdtempSync(join(tmpdir(), "diffinery-registry-"));
24+let base;
25+let host;
26+let dev;
27+let devLog = "";
28+let token;
29+let owner;
30+
31+const freePort = () =>
32+ new Promise((res) => {
33+ const s = createServer();
34+ s.listen(0, "127.0.0.1", () => {
35+ const { port } = s.address();
36+ s.close(() => res(port));
37+ });
38+ });
39+const sleep = (ms) => new Promise((r) => setTimeout(r, ms));
40+async function until(what, fn, timeoutMs = 60_000) {
41+ const end = Date.now() + timeoutMs;
42+ while (Date.now() < end) {
43+ const v = await fn();
44+ if (v) return v;
45+ await sleep(500);
46+ }
47+ assert.fail(`timed out waiting for ${what}\n--- wrangler log tail ---\n${devLog.slice(-4000)}`);
48+}
49+
50+const sha = (b) => `sha256:${createHash("sha256").update(b).digest("hex")}`;
51+const auth = (tok = token) => ({ authorization: `Basic ${Buffer.from(`anyone:${tok}`).toString("base64")}` });
52+
53+/** A registry call: path is below /v2/. */
54+async function reg(method, path, { body, headers = {}, tok } = {}) {
55+ const res = await fetch(`${base}/v2/${path}`, { method, body, headers: { ...auth(tok), ...headers }, redirect: "manual", duplex: body instanceof ReadableStream ? "half" : undefined });
56+ const buf = Buffer.from(await res.arrayBuffer());
57+ let json = null;
58+ try {
59+ json = buf.length ? JSON.parse(buf.toString("utf8")) : null;
60+ } catch {}
61+ return { status: res.status, headers: res.headers, body: buf, json };
62+}
63+
64+async function api(method, path, body, tok = token) {
65+ const res = await fetch(`${base}/api/v3${path}`, { method, headers: { authorization: `Bearer ${tok}`, "content-type": "application/json" }, body: body === undefined ? undefined : JSON.stringify(body) });
66+ const text = await res.text();
67+ let json = null;
68+ try {
69+ json = text ? JSON.parse(text) : null;
70+ } catch {
71+ json = text;
72+ }
73+ return { status: res.status, json };
74+}
75+async function ok(method, path, body, tok) {
76+ const r = await api(method, path, body, tok);
77+ assert.ok(r.status >= 200 && r.status < 300, `${method} ${path} -> ${r.status} ${JSON.stringify(r.json)}`);
78+ return r.json;
79+}
80+
81+/** Uploads a blob in one POST. */
82+async function pushBlob(name, bytes, tok) {
83+ const r = await reg("POST", `${name}/blobs/uploads/?digest=${sha(bytes)}`, { body: bytes, headers: { "content-type": "application/octet-stream" }, tok });
84+ assert.equal(r.status, 201, JSON.stringify(r.json));
85+ return { digest: sha(bytes), size: bytes.length };
86+}
87+
88+/** Pushes a small single-layer image and returns its manifest digest. */
89+async function pushImage(name, tag, { layer = randomBytes(2048), labels = {}, tok } = {}) {
90+ const config = Buffer.from(JSON.stringify({ architecture: "amd64", os: "linux", config: { Labels: labels }, rootfs: { type: "layers", diff_ids: [sha(layer)] } }));
91+ const c = await pushBlob(name, config, tok);
92+ const l = await pushBlob(name, layer, tok);
93+ const manifest = Buffer.from(
94+ JSON.stringify({
95+ schemaVersion: 2,
96+ mediaType: "application/vnd.oci.image.manifest.v1+json",
97+ config: { mediaType: "application/vnd.oci.image.config.v1+json", ...c },
98+ layers: [{ mediaType: "application/vnd.oci.image.layer.v1.tar+gzip", ...l }],
99+ }),
100+ );
101+ const r = await reg("PUT", `${name}/manifests/${tag}`, { body: manifest, headers: { "content-type": "application/vnd.oci.image.manifest.v1+json" }, tok });
102+ assert.equal(r.status, 201, JSON.stringify(r.json));
103+ assert.equal(r.headers.get("docker-content-digest"), sha(manifest));
104+ return { digest: sha(manifest), manifest, layer, size: manifest.length };
105+}
106+
107+before(async () => {
108+ assert.ok(existsSync(AGENT), `build the runner first: pnpm --filter @diffinery/runner build (missing ${AGENT})`);
109+ const port = await freePort();
110+ base = `http://127.0.0.1:${port}`;
111+ host = `127.0.0.1:${port}`;
112+ const persist = join(work, "state");
113+ execFileSync("pnpm", ["exec", "wrangler", "d1", "migrations", "apply", "diffinery-dev", "--local", "-e", "dev", "--persist-to", persist], { cwd: APP, stdio: "pipe", env: { ...process.env, CI: "1" } });
114+ dev = spawn("pnpm", ["exec", "wrangler", "dev", "-e", "dev", "--ip", "127.0.0.1", "--port", String(port), "--persist-to", persist, "--var", `PUBLIC_URL:${base}`, "--var", `MASTER_KEY:${randomBytes(32).toString("base64")}`, "--show-interactive-dev-session=false"], {
115+ cwd: APP,
116+ env: { ...process.env, CI: "1", WRANGLER_SEND_METRICS: "false" },
117+ stdio: ["ignore", "pipe", "pipe"],
118+ detached: true,
119+ });
120+ dev.stdout.on("data", (d) => (devLog += d));
121+ dev.stderr.on("data", (d) => (devLog += d));
122+ await until("wrangler dev", async () => {
123+ try {
124+ return (await fetch(`${base}/healthz`)).ok;
125+ } catch {
126+ return false;
127+ }
128+ }, 120_000);
129+ const res = await fetch(`${base}/api/v3/diffinery/tokens`, { method: "POST", headers: { "content-type": "application/json" }, body: JSON.stringify({ name: "registry-e2e", scopes: ["admin"] }) });
130+ assert.equal(res.status, 201, await res.clone().text());
131+ token = (await res.json()).token;
132+ owner = (await ok("GET", "/user")).login;
133+});
134+
135+after(() => {
136+ if (dev?.pid) {
137+ try {
138+ process.kill(-dev.pid, "SIGTERM");
139+ } catch {}
140+ }
141+ if (process.env.E2E_KEEP !== "1") rmSync(work, { recursive: true, force: true });
142+});
143+
144+test("registry: the OCI distribution API", async () => {
145+ const v2 = await reg("GET", "");
146+ assert.equal(v2.status, 200);
147+ assert.equal(v2.headers.get("docker-distribution-api-version"), "registry/2.0");
148+ // Unknown tokens are refused with a Basic challenge so `docker login` reports it.
149+ const bad = await fetch(`${base}/v2/`, { headers: auth("dfy_nope") });
150+ assert.equal(bad.status, 401);
151+ assert.match(bad.headers.get("www-authenticate") ?? "", /^Basic realm=/);
152+
153+ const name = `${owner}/tools/api`;
154+ assert.equal((await reg("GET", `${owner}/bad..name/blobs/${sha("x")}`)).json.errors[0].code, "NAME_INVALID");
155+ assert.equal((await reg("GET", `${name}/manifests/latest`)).status, 404);
156+
157+ // Monolithic upload, then HEAD and ranged GET.
158+ const blob = randomBytes(5000);
159+ const { digest } = await pushBlob(name, blob);
160+ const head = await reg("HEAD", `${name}/blobs/${digest}`);
161+ assert.equal(head.status, 200);
162+ assert.equal(head.headers.get("content-length"), "5000");
163+ assert.equal(head.headers.get("docker-content-digest"), digest);
164+ const part = await reg("GET", `${name}/blobs/${digest}`, { headers: { range: "bytes=100-199" } });
165+ assert.equal(part.status, 206);
166+ assert.deepEqual(part.body, blob.subarray(100, 200));
167+ assert.equal(part.headers.get("content-range"), "bytes 100-199/5000");
168+
169+ // Chunked upload: three PATCHes with Content-Range, then PUT with the last bytes.
170+ const big = randomBytes(23 * 1024 * 1024 + 123);
171+ const start = await reg("POST", `${name}/blobs/uploads/`);
172+ assert.equal(start.status, 202);
173+ let loc = start.headers.get("location");
174+ assert.match(loc, /\/blobs\/uploads\/[0-9a-f]{32}$/);
175+ const cuts = [0, 9 * 1024 * 1024, 11 * 1024 * 1024, 22 * 1024 * 1024, big.length];
176+ for (let i = 0; i < 3; i++) {
177+ const chunk = big.subarray(cuts[i], cuts[i + 1]);
178+ const r = await fetch(`${base}${loc}`, { method: "PATCH", body: chunk, headers: { ...auth(), "content-type": "application/octet-stream", "content-range": `${cuts[i]}-${cuts[i + 1] - 1}` } });
179+ assert.equal(r.status, 202, await r.text());
180+ assert.equal(r.headers.get("range"), `0-${cuts[i + 1] - 1}`);
181+ loc = r.headers.get("location");
182+ }
183+ // A chunk that doesn't start where the upload is gets 416 and the current range.
184+ const wrong = await fetch(`${base}${loc}`, { method: "PATCH", body: Buffer.from("zz"), headers: { ...auth(), "content-range": "5-6" } });
185+ assert.equal(wrong.status, 416);
186+ assert.equal(wrong.headers.get("range"), `0-${cuts[3] - 1}`);
187+ const status = await fetch(`${base}${loc}`, { headers: auth() });
188+ assert.equal(status.status, 204);
189+ assert.equal(status.headers.get("diffinery-upload-size"), String(cuts[3]));
190+ const done = await fetch(`${base}${loc}?digest=${sha(big)}`, { method: "PUT", body: big.subarray(cuts[3]), headers: { ...auth(), "content-type": "application/octet-stream", "content-range": `${cuts[3]}-${big.length - 1}` } });
191+ assert.equal(done.status, 201, await done.text());
192+ const back = await reg("GET", `${name}/blobs/${sha(big)}`);
193+ assert.equal(back.body.length, big.length);
194+ assert.ok(back.body.equals(big), "the chunked blob comes back intact");
195+
196+ // A streamed single PATCH (how docker sends layers), larger than one R2 part.
197+ const streamed = randomBytes(12 * 1024 * 1024 + 7);
198+ const s2 = await reg("POST", `${name}/blobs/uploads/`);
199+ const pieces = [streamed.subarray(0, 1_000_000), streamed.subarray(1_000_000)];
200+ const stream = new ReadableStream({ pull(ctl) { const p = pieces.shift(); p ? ctl.enqueue(p) : ctl.close(); } });
201+ const p2 = await fetch(`${base}${s2.headers.get("location")}`, { method: "PATCH", body: stream, duplex: "half", headers: auth() });
202+ assert.equal(p2.status, 202);
203+ assert.equal((await fetch(`${base}${p2.headers.get("location")}?digest=${sha(streamed)}`, { method: "PUT", headers: auth() })).status, 201);
204+ assert.ok((await reg("GET", `${name}/blobs/${sha(streamed)}`)).body.equals(streamed));
205+
206+ // A wrong digest is refused and the upload is gone.
207+ const s3 = await reg("POST", `${name}/blobs/uploads/`);
208+ const badPut = await fetch(`${base}${s3.headers.get("location")}?digest=${sha("other")}`, { method: "PUT", body: Buffer.from("content"), headers: auth() });
209+ assert.equal(badPut.status, 400);
210+ assert.equal((await badPut.json()).errors[0].code, "DIGEST_INVALID");
211+ assert.equal((await fetch(`${base}${s3.headers.get("location")}`, { headers: auth() })).status, 404);
212+
213+ // Manifests must only use blobs the package has.
214+ const orphan = Buffer.from(JSON.stringify({ schemaVersion: 2, mediaType: "application/vnd.oci.image.manifest.v1+json", config: { mediaType: "application/vnd.oci.image.config.v1+json", digest: sha("missing"), size: 7 }, layers: [] }));
215+ const refused = await reg("PUT", `${name}/manifests/broken`, { body: orphan, headers: { "content-type": "application/vnd.oci.image.manifest.v1+json" } });
216+ assert.equal(refused.status, 400);
217+ assert.equal(refused.json.errors[0].code, "MANIFEST_BLOB_UNKNOWN");
218+
219+ // Images by tag and digest, with an index on top.
220+ const amd = await pushImage(name, "amd64");
221+ const index = Buffer.from(
222+ JSON.stringify({
223+ schemaVersion: 2,
224+ mediaType: "application/vnd.oci.image.index.v1+json",
225+ manifests: [{ mediaType: "application/vnd.oci.image.manifest.v1+json", digest: amd.digest, size: amd.size, platform: { os: "linux", architecture: "amd64" } }],
226+ }),
227+ );
228+ assert.equal((await reg("PUT", `${name}/manifests/v1`, { body: index, headers: { "content-type": "application/vnd.oci.image.index.v1+json" } })).status, 201);
229+ for (const ref of ["v1", sha(index)]) {
230+ const m = await reg("GET", `${name}/manifests/${ref}`, { headers: { accept: "application/vnd.oci.image.index.v1+json" } });
231+ assert.equal(m.status, 200);
232+ assert.equal(m.headers.get("content-type"), "application/vnd.oci.image.index.v1+json");
233+ assert.equal(m.headers.get("docker-content-digest"), sha(index));
234+ assert.ok(m.body.equals(index));
235+ }
236+ assert.equal((await reg("HEAD", `${name}/manifests/amd64`)).headers.get("docker-content-digest"), amd.digest);
237+ // Pushing by digest checks the digest.
238+ assert.equal((await reg("PUT", `${name}/manifests/${sha("nope")}`, { body: index, headers: { "content-type": "application/vnd.oci.image.index.v1+json" } })).json.errors[0].code, "DIGEST_INVALID");
239+
240+ // Tags, paginated.
241+ await pushImage(name, "v2");
242+ assert.deepEqual((await reg("GET", `${name}/tags/list`)).json, { name, tags: ["amd64", "v1", "v2"] });
243+ const page1 = await reg("GET", `${name}/tags/list?n=2`);
244+ assert.deepEqual(page1.json.tags, ["amd64", "v1"]);
245+ assert.match(page1.headers.get("link"), /last=v1>; rel="next"/);
246+ assert.deepEqual((await reg("GET", `${name}/tags/list?n=2&last=v1`)).json.tags, ["v2"]);
247+
248+ // Referrers: an artifact whose subject is the amd64 image.
249+ const empty = await pushBlob(name, Buffer.from("{}"));
250+ const sig = Buffer.from(
251+ JSON.stringify({
252+ schemaVersion: 2,
253+ mediaType: "application/vnd.oci.image.manifest.v1+json",
254+ artifactType: "application/vnd.example.signature",
255+ config: { mediaType: "application/vnd.oci.empty.v1+json", ...empty },
256+ layers: [],
257+ subject: { mediaType: "application/vnd.oci.image.manifest.v1+json", digest: amd.digest, size: amd.size },
258+ annotations: { "org.example.signed-by": "e2e" },
259+ }),
260+ );
261+ const sigPut = await reg("PUT", `${name}/manifests/${sha(sig)}`, { body: sig, headers: { "content-type": "application/vnd.oci.image.manifest.v1+json" } });
262+ assert.equal(sigPut.status, 201);
263+ assert.equal(sigPut.headers.get("oci-subject"), amd.digest);
264+ const refs = await reg("GET", `${name}/referrers/${amd.digest}`);
265+ assert.equal(refs.json.manifests.length, 1);
266+ assert.equal(refs.json.manifests[0].artifactType, "application/vnd.example.signature");
267+ assert.equal((await reg("GET", `${name}/referrers/${amd.digest}?artifactType=other`)).json.manifests.length, 0);
268+
269+ // Cross-repository mount, and a mount that falls back to a normal upload.
270+ const other = `${owner}/tools/other`;
271+ const mounted = await reg("POST", `${other}/blobs/uploads/?mount=${digest}&from=${name}`);
272+ assert.equal(mounted.status, 201);
273+ assert.equal((await reg("HEAD", `${other}/blobs/${digest}`)).status, 200);
274+ assert.equal((await reg("POST", `${other}/blobs/uploads/?mount=${sha("never")}&from=${name}`)).status, 202);
275+ // The blob isn't readable through a package that doesn't hold it.
276+ assert.equal((await reg("HEAD", `${other}/blobs/${sha(big)}`)).status, 404);
277+
278+ // Deleting a tag keeps the manifest; deleting the manifest removes it and its tags.
279+ assert.equal((await reg("DELETE", `${name}/manifests/v2`)).status, 202);
280+ assert.equal((await reg("GET", `${name}/manifests/v2`)).status, 404);
281+ assert.equal((await reg("DELETE", `${name}/manifests/${amd.digest}`)).status, 202);
282+ assert.equal((await reg("GET", `${name}/manifests/amd64`)).status, 404);
283+
284+ const catalog = await reg("GET", "_catalog");
285+ assert.ok(catalog.json.repositories.includes(name) && catalog.json.repositories.includes(other));
286+});
287+
288+test("registry: token scopes, the packages API and repository links", async () => {
289+ const read = (await ok("POST", "/diffinery/tokens", { name: "pull only", scopes: ["read"] })).token;
290+ const name = `${owner}/scoped`;
291+ const img = await pushImage(name, "latest");
292+ assert.equal((await reg("GET", `${name}/manifests/latest`, { tok: read })).status, 200);
293+ const denied = await reg("POST", `${name}/blobs/uploads/`, { tok: read });
294+ assert.equal(denied.status, 403);
295+ assert.equal(denied.json.errors[0].code, "DENIED");
296+ assert.equal((await reg("DELETE", `${name}/manifests/latest`, { tok: read })).status, 403);
297+
298+ // GitHub-shaped packages API.
299+ const list = await ok("GET", "/user/packages?package_type=container");
300+ const pkg = list.find((p) => p.name === "scoped");
301+ assert.ok(pkg, JSON.stringify(list));
302+ assert.equal(pkg.visibility, "private");
303+ assert.equal(pkg.image, `${host}/${owner}/scoped`);
304+ const versions = await ok("GET", "/user/packages/container/scoped/versions");
305+ assert.equal(versions.length, 1);
306+ assert.equal(versions[0].name, img.digest);
307+ assert.deepEqual(versions[0].metadata.container.tags, ["latest"]);
308+ assert.equal((await ok("PATCH", "/user/packages/container/scoped", { visibility: "public" })).visibility, "public");
309+ assert.equal((await api("PATCH", "/user/packages/container/scoped", { visibility: "secret" })).status, 422);
310+ assert.equal((await api("DELETE", `/user/packages/container/scoped/versions/${versions[0].id}`)).status, 204);
311+ assert.equal((await reg("GET", `${name}/manifests/latest`)).status, 404);
312+ assert.equal((await api("DELETE", "/user/packages/container/scoped")).status, 204);
313+ assert.equal((await api("GET", "/user/packages/container/scoped")).status, 404);
314+
315+ // diffinery.com/<owner>/<repo>/... is linked to the repository when the pusher can write to it.
316+ await ok("POST", "/user/repos", { name: "linked" });
317+ await pushImage(`${owner}/linked/api`, "v1");
318+ assert.equal((await ok("GET", "/user/packages/container/linked%2Fapi")).repository.full_name, `${owner}/linked`);
319+ const repoPkgs = await ok("GET", `/repos/${owner}/linked/packages`);
320+ assert.deepEqual(repoPkgs.map((p) => p.name), ["linked/api"]);
321+ // ...and so is an image whose org.opencontainers.image.source label names one.
322+ await ok("POST", "/user/repos", { name: "labelled" });
323+ await pushImage(`${owner}/some-image`, "v1", { labels: { "org.opencontainers.image.source": `${base}/${owner}/labelled` } });
324+ assert.equal((await ok("GET", "/user/packages/container/some-image")).repository.full_name, `${owner}/labelled`);
325+ // The web pages render.
326+ for (const path of [`/${owner}/-/packages`, `/${owner}/-/packages/container/linked/api`, `/${owner}/linked/packages`]) {
327+ const page = await fetch(`${base}${path}`);
328+ assert.equal(page.status, 200, path);
329+ assert.match(await page.text(), /linked\/api|some-image/);
330+ }
331+});
332+
333+function git(cwd, args) {
334+ const a = Buffer.from(`x-token:${token}`).toString("base64");
335+ return execFileSync("git", ["-c", `http.extraHeader=Authorization: Basic ${a}`, "-c", "protocol.version=2", ...args], {
336+ cwd,
337+ encoding: "utf8",
338+ stdio: ["ignore", "pipe", "pipe"],
339+ env: { ...process.env, GIT_TERMINAL_PROMPT: "0", GIT_AUTHOR_NAME: "E2E", GIT_AUTHOR_EMAIL: "e2e@example.com", GIT_COMMITTER_NAME: "E2E", GIT_COMMITTER_EMAIL: "e2e@example.com" },
340+ });
341+}
342+
343+function spawnAsync(cmd, args, env, timeoutMs) {
344+ return new Promise((done) => {
345+ const child = spawn(cmd, args, { env: { ...process.env, ...env } });
346+ let out = "";
347+ child.stdout.on("data", (d) => (out += d));
348+ child.stderr.on("data", (d) => (out += d));
349+ const t = setTimeout(() => child.kill("SIGKILL"), timeoutMs);
350+ child.on("close", (status) => {
351+ clearTimeout(t);
352+ done({ status, out });
353+ });
354+ });
355+}
356+
357+async function runOneJob(repo) {
358+ const claim = await until("a job to claim", async () => {
359+ const r = await api("POST", `/repos/${owner}/${repo}/actions/runners/claim`, { name: "e2e", labels: ["linux"] });
360+ return r.status === 200 ? r.json : null;
361+ });
362+ const r = await spawnAsync(process.execPath, [AGENT], { DIFFINERY_API_URL: claim.api_url, DIFFINERY_JOB_ID: claim.job_id, DIFFINERY_JOB_TOKEN: claim.token, DIFFINERY_WORK_ROOT: join(work, `job-${claim.job_id}`) }, 180_000);
363+ const log = await (await fetch(`${base}/api/v3/repos/${owner}/${repo}/actions/jobs/${claim.job_id}/logs`, { headers: { authorization: `Bearer ${token}` } })).text();
364+ return { name: claim.name, status: r.status, out: r.out, log };
365+}
366+
367+test("registry: a workflow's GITHUB_TOKEN follows the packages permission", async () => {
368+ await ok("POST", "/user/repos", { name: "shipyard" });
369+ const src = join(work, "shipyard");
370+ mkdirSync(join(src, ".github/workflows"), { recursive: true });
371+ // Push a tiny image with curl, the way any registry client would.
372+ writeFileSync(
373+ join(src, "push.sh"),
374+ [
375+ 'R="$GITHUB_SERVER_URL/v2/$GITHUB_REPOSITORY_OWNER/shipyard"',
376+ `printf '%s' '{"architecture":"amd64","os":"linux","rootfs":{"type":"layers","diff_ids":[]}}' > cfg.json`,
377+ `d="sha256:$(sha256sum cfg.json | cut -d' ' -f1)"`,
378+ 'curl -sS -o /dev/null -w "blob=%{http_code}\\n" -u "x:$GITHUB_TOKEN" -X POST "$R/blobs/uploads/?digest=$d" --data-binary @cfg.json',
379+ `printf '{"schemaVersion":2,"mediaType":"application/vnd.oci.image.manifest.v1+json","config":{"mediaType":"application/vnd.oci.image.config.v1+json","digest":"%s","size":%s},"layers":[]}' "$d" "$(stat -c %s cfg.json)" > m.json`,
380+ 'curl -sS -o /dev/null -w "$1=%{http_code}\\n" -u "x:$GITHUB_TOKEN" -X PUT "$R/manifests/$1" -H "content-type: application/vnd.oci.image.manifest.v1+json" --data-binary @m.json',
381+ "",
382+ ].join("\n"),
383+ );
384+ writeFileSync(
385+ join(src, ".github/workflows/images.yml"),
386+ `on: push
387+env:
388+ GITHUB_TOKEN: \${{ secrets.GITHUB_TOKEN }}
389+jobs:
390+ publish:
391+ runs-on: [self-hosted, linux]
392+ permissions:
393+ packages: write
394+ steps:
395+ - uses: actions/checkout@v4
396+ - run: bash push.sh ci
397+ reader:
398+ needs: publish
399+ runs-on: [self-hosted, linux]
400+ permissions:
401+ packages: read
402+ steps:
403+ - uses: actions/checkout@v4
404+ - run: |
405+ curl -sS -o /dev/null -w "pull=%{http_code}\\n" -u "x:$GITHUB_TOKEN" "$GITHUB_SERVER_URL/v2/$GITHUB_REPOSITORY_OWNER/shipyard/manifests/ci" -H "accept: application/vnd.oci.image.manifest.v1+json"
406+ bash push.sh nope
407+ outsider:
408+ needs: reader
409+ runs-on: [self-hosted, linux]
410+ permissions:
411+ contents: read
412+ steps:
413+ - uses: actions/checkout@v4
414+ - run: bash push.sh nope
415+`,
416+ );
417+ git(src, ["init", "-q", "-b", "main"]);
418+ git(src, ["add", "-A"]);
419+ git(src, ["commit", "-q", "-m", "images"]);
420+ git(src, ["remote", "add", "origin", `${base}/${owner}/shipyard.git`]);
421+ git(src, ["push", "-q", "origin", "main"]);
422+
423+ const publish = await runOneJob("shipyard");
424+ assert.equal(publish.name, "publish");
425+ assert.match(publish.log, /blob=201/, publish.log);
426+ assert.match(publish.log, /ci=201/, publish.log);
427+ // Created by the workflow, so linked to its repository.
428+ const pkg = await ok("GET", "/user/packages/container/shipyard");
429+ assert.equal(pkg.repository.full_name, `${owner}/shipyard`);
430+ assert.equal((await ok("GET", "/user/packages/container/shipyard/versions"))[0].pushed_by, "diffinery-actions[bot]");
431+
432+ const reader = await runOneJob("shipyard");
433+ assert.equal(reader.name, "reader");
434+ assert.match(reader.log, /pull=200/, reader.log);
435+ assert.match(reader.log, /blob=403/, reader.log);
436+ assert.match(reader.log, /nope=403/, reader.log);
437+
438+ const outsider = await runOneJob("shipyard");
439+ assert.equal(outsider.name, "outsider");
440+ assert.match(outsider.log, /blob=404/, outsider.log);
441+ assert.deepEqual((await reg("GET", `${owner}/shipyard/tags/list`)).json.tags, ["ci"]);
442+});
443+
444+const hasDocker = spawnSync("docker", ["info"], { stdio: "ignore", timeout: 15_000 }).status === 0;
445+
446+test("registry: docker login, push and pull", { skip: !hasDocker && "Docker is not available" }, async () => {
447+ const dir = join(work, "image");
448+ mkdirSync(dir, { recursive: true });
449+ const payload = randomBytes(6 * 1024 * 1024);
450+ writeFileSync(join(dir, "payload.bin"), payload);
451+ writeFileSync(join(dir, "Dockerfile"), 'FROM scratch\nCOPY payload.bin /payload.bin\nLABEL org.opencontainers.image.description="e2e image"\n');
452+ const image = `${host}/${owner}/dockerized:v1`;
453+ const config = join(work, "docker-config");
454+ mkdirSync(config, { recursive: true });
455+ // Run docker without blocking the event loop: undici would keep sockets the server already closed.
456+ const docker = async (args, input) => {
457+ const r = await new Promise((done) => {
458+ const child = spawn("docker", ["--config", config, ...args], { cwd: dir });
459+ let out = "";
460+ let err = "";
461+ child.stdout.on("data", (d) => (out += d));
462+ child.stderr.on("data", (d) => (err += d));
463+ child.on("close", (status) => done({ status, out, err }));
464+ child.stdin.end(input ?? "");
465+ });
466+ assert.equal(r.status, 0, `docker ${args.join(" ")}: ${r.err}${r.out}`);
467+ return r.out;
468+ };
469+ assert.match(await docker(["login", host, "-u", "e2e", "--password-stdin"], token), /Login Succeeded/);
470+ await docker(["build", "-q", "-t", image, "."]);
471+ await docker(["push", image]);
472+ await docker(["rmi", image]);
473+ await docker(["pull", "-q", image]);
474+ const id = (await docker(["create", image, "true"])).trim();
475+ try {
476+ await docker(["cp", `${id}:/payload.bin`, join(work, "payload.out")]);
477+ } finally {
478+ await docker(["rm", id]);
479+ await docker(["rmi", image]);
480+ }
481+ assert.ok(readFileSync(join(work, "payload.out")).equals(payload), "the pulled image holds the same bytes");
482+ const pkg = await ok("GET", "/user/packages/container/dockerized");
483+ assert.equal(pkg.description, "e2e image");
484+});
modifiedcontainers/runner/image/docker+78−1View file
@@ -2,15 +2,60 @@
22 # Hosted Diffinery runners: starts the Docker daemon the first time anything needs
33 # it (a step running `docker`, job containers, services, Docker actions), then runs
44 # the real CLI. The daemon runs in its own session so step cleanup leaves it alone.
5+#
6+# The daemon reaches the Diffinery container registry through the runner's registry
7+# relay (agent.mjs --registry-relay), which splits image layers into chunks under
8+# Cloudflare's 100 MB request limit. Builders that `docker buildx create` starts
9+# (docker/setup-buildx-action) are pointed at the relay too.
510 real=/usr/bin/docker
611 ready=/run/diffinery-dockerd.ready
12+state=/run/diffinery
13+relay_port=3129
14+sys_path=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
15+
16+# The Diffinery host this runner reports to, when it is reached over HTTPS.
17+registry_host() {
18+ local url=${DIFFINERY_API_URL:-}
19+ [[ "$url" == https://* ]] || return 1
20+ url=${url#https://}
21+ echo "${url%%/*}"
22+}
23+
24+start_relay() {
25+ local host=$1
26+ mkdir -p "$state" "/etc/docker/certs.d/$host" || return 1
27+ # A certificate for the Diffinery host that only this daemon (and its builders) trust.
28+ openssl req -x509 -newkey ec -pkeyopt ec_paramgen_curve:prime256v1 -nodes -days 30 -subj "/CN=$host" \
29+ -addext "subjectAltName=DNS:$host" -keyout "$state/relay.key" -out "$state/relay.crt" >/dev/null 2>&1 || return 1
30+ cp "$state/relay.crt" "/etc/docker/certs.d/$host/ca.crt"
31+ printf '[registry."%s"]\n ca = ["%s"]\n' "$host" "$state/relay.crt" >"$state/buildkitd.toml"
32+ env -i PATH="$sys_path" DIFFINERY_RELAY_HOST="$host" DIFFINERY_RELAY_UPSTREAM="https://$host" DIFFINERY_RELAY_PORT="$relay_port" \
33+ DIFFINERY_RELAY_CERT="$state/relay.crt" DIFFINERY_RELAY_KEY="$state/relay.key" \
34+ setsid -f node /opt/diffinery/agent.mjs --registry-relay >/var/log/diffinery-relay.log 2>&1 </dev/null
35+ for _ in $(seq 1 80); do
36+ if (exec 3<>"/dev/tcp/127.0.0.1/$relay_port") 2>/dev/null; then
37+ echo "$host" >"$state/relay.host"
38+ return 0
39+ fi
40+ sleep 0.25
41+ done
42+ return 1
43+}
44+
745 if [ ! -e "$ready" ] && [ -z "${DOCKER_HOST:-}" ] && [ "$(id -u)" = 0 ]; then
846 exec 9>/run/diffinery-dockerd.lock
947 flock 9
1048 if [ ! -e "$ready" ]; then
1149 if ! "$real" version >/dev/null 2>&1; then
1250 echo "Starting the Docker daemon" >&2
13- env -i PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin setsid -f dockerd >/var/log/dockerd.log 2>&1 </dev/null
51+ proxy=()
52+ if host=$(registry_host) && start_relay "$host"; then
53+ proxy=(HTTPS_PROXY="http://127.0.0.1:$relay_port")
54+ elif [ -n "${host:-}" ]; then
55+ echo "The registry relay did not start, so image layers over 100 MB can't be pushed to $host:" >&2
56+ tail -n 5 /var/log/diffinery-relay.log >&2 2>/dev/null
57+ fi
58+ env -i PATH="$sys_path" "${proxy[@]}" setsid -f dockerd >/var/log/dockerd.log 2>&1 </dev/null
1459 for _ in $(seq 1 240); do "$real" version >/dev/null 2>&1 && break; sleep 0.25; done
1560 "$real" version >/dev/null 2>&1 || { echo "The Docker daemon did not start:" >&2; tail -n 20 /var/log/dockerd.log >&2; exit 1; }
1661 fi
@@ -18,4 +63,36 @@
1863 fi
1964 exec 9>&-
2065 fi
66+
67+# docker buildx create (docker-container driver): run BuildKit on the host network with
68+# the relay as its HTTPS proxy, trusting the relay for the Diffinery host.
69+if [ "${1:-}" = buildx ] && [ "${2:-}" = create ] && [ -s "$state/relay.host" ]; then
70+ host=$(cat "$state/relay.host")
71+ args=("$@")
72+ driver=docker-container
73+ cfg=-1
74+ for ((i = 2; i < ${#args[@]}; i++)); do
75+ case "${args[i]}" in
76+ --driver) driver=${args[i + 1]:-} ;;
77+ --driver=*) driver=${args[i]#--driver=} ;;
78+ --buildkitd-config | --config) cfg=$((i + 1)) ;;
79+ --driver-opt) [[ "${args[i + 1]:-}" == network=* ]] && driver=custom-network ;;
80+ --driver-opt=network=*) driver=custom-network ;;
81+ esac
82+ done
83+ if [ "$driver" = docker-container ]; then
84+ if [ "$cfg" -ge 0 ] && [ -f "${args[cfg]}" ]; then
85+ if ! grep -qF "registry.\"$host\"" "${args[cfg]}"; then
86+ merged=$(mktemp "$state/buildkitd-XXXXXX.toml")
87+ { cat "${args[cfg]}"; echo; cat "$state/buildkitd.toml"; } >"$merged"
88+ args[cfg]=$merged
89+ fi
90+ elif [ "$cfg" -lt 0 ]; then
91+ args+=(--buildkitd-config "$state/buildkitd.toml")
92+ fi
93+ args+=(--driver-opt network=host --driver-opt "env.HTTPS_PROXY=http://127.0.0.1:$relay_port" --driver-opt "env.https_proxy=http://127.0.0.1:$relay_port")
94+ exec "$real" "${args[@]}"
95+ fi
96+fi
97+
2198 exec "$real" "$@"
modifiedcontainers/runner/src/main.ts+13−1View file
@@ -5,27 +5,32 @@
55 * agent.mjs --spec job.json run a JobSpec from a file (reports to spec.apiUrl,
66 * or DIFFINERY_API_URL; `--offline` logs to stdout only)
77 * agent.mjs --sandbox AI-agent sandbox HTTP server on :8080
8+ * agent.mjs --registry-relay HTTPS proxy for the Docker daemon that splits big
9+ * image layer uploads to the Diffinery registry (see relay.ts)
810 */
911 import { readFile } from "node:fs/promises";
1012 import type { JobSpec } from "@diffinery/shared";
1113 import { apiBaseFor, runAgentJob } from "./agent.js";
1214 import { RunnerApi } from "./api.js";
1315 import { startSandbox } from "./sandbox.js";
16+import { startRelay } from "./relay.js";
1417
1518 interface Args {
1619 spec?: string;
1720 sandbox: boolean;
21+ relay: boolean;
1822 offline: boolean;
1923 workRoot?: string;
2024 }
2125
2226 function parseArgs(argv: string[]): Args {
23- const a: Args = { sandbox: false, offline: false };
27+ const a: Args = { sandbox: false, relay: false, offline: false };
2428 for (let i = 0; i < argv.length; i++) {
2529 const v = argv[i]!;
2630 if (v === "--spec") a.spec = argv[++i];
2731 else if (v.startsWith("--spec=")) a.spec = v.slice(7);
2832 else if (v === "--sandbox") a.sandbox = true;
33+ else if (v === "--registry-relay") a.relay = true;
2934 else if (v === "--offline") a.offline = true;
3035 else if (v === "--work-root") a.workRoot = argv[++i];
3136 else if (v === "--help" || v === "-h") {
@@ -46,6 +51,13 @@
4651 const env = process.env;
4752 const workRoot = args.workRoot ?? env.DIFFINERY_WORK_ROOT ?? "/workspace";
4853
54+ if (args.relay) {
55+ const { DIFFINERY_RELAY_HOST: host, DIFFINERY_RELAY_UPSTREAM: upstream, DIFFINERY_RELAY_CERT: cert, DIFFINERY_RELAY_KEY: key } = env;
56+ if (!host || !upstream || !cert || !key) fail("DIFFINERY_RELAY_HOST, DIFFINERY_RELAY_UPSTREAM, DIFFINERY_RELAY_CERT and DIFFINERY_RELAY_KEY are required with --registry-relay");
57+ await startRelay({ port: Number(env.DIFFINERY_RELAY_PORT ?? 3129), host, upstream, chunkSize: env.DIFFINERY_RELAY_CHUNK ? Number(env.DIFFINERY_RELAY_CHUNK) : undefined, cert: await readFile(cert, "utf8"), key: await readFile(key, "utf8") });
58+ return;
59+ }
60+
4961 if (args.sandbox) {
5062 const secret = env.DIFFINERY_SANDBOX_SECRET;
5163 if (!secret) fail("DIFFINERY_SANDBOX_SECRET is required in --sandbox mode");
addedcontainers/runner/src/relay.ts+235−0View file
@@ -1,0 +1,235 @@
1+import http from "node:http";
2+import https from "node:https";
3+import net from "node:net";
4+import tls from "node:tls";
5+
6+/**
7+ * Registry relay for hosted runners: an HTTP(S) proxy for the Docker daemon.
8+ *
9+ * Cloudflare refuses request bodies over 100 MB, and Docker sends each image layer in a
10+ * single request, so a big layer could never reach the Diffinery registry directly. The
11+ * Docker daemon on a hosted runner uses this relay as its HTTPS proxy. Connections to the
12+ * Diffinery host are terminated here (the daemon trusts this relay's certificate for that
13+ * host only, through /etc/docker/certs.d), and blob uploads are re-sent as a series of
14+ * chunked PATCH requests with Content-Range, which the registry API supports. Everything
15+ * else is passed through unchanged, and other hosts are tunnelled untouched.
16+ */
17+export interface RelayOptions {
18+ /** Port to listen on (127.0.0.1). */
19+ port: number;
20+ /** Host name the relay intercepts, e.g. diffinery.com. */
21+ host: string;
22+ /** Where intercepted requests really go, e.g. https://diffinery.com. */
23+ upstream: string;
24+ cert: string;
25+ key: string;
26+ /** Upload chunk size; must stay under Cloudflare's 100 MB limit. */
27+ chunkSize?: number;
28+ log?: (line: string) => void;
29+}
30+
31+const HOP_BY_HOP = new Set(["connection", "keep-alive", "proxy-authenticate", "proxy-authorization", "proxy-connection", "te", "trailer", "transfer-encoding", "upgrade", "host", "content-length"]);
32+const UPLOAD_PATH = /^\/v2\/.+\/blobs\/uploads\/[^/?]+(\?.*)?$/;
33+
34+export interface Relay {
35+ port: number;
36+ close(): Promise<void>;
37+}
38+
39+export async function startRelay(o: RelayOptions): Promise<Relay> {
40+ const log = o.log ?? ((l: string) => process.stdout.write(`${l}\n`));
41+ const upstream = new URL(o.upstream);
42+ const client = upstream.protocol === "https:" ? https : http;
43+ const agent = new client.Agent({ keepAlive: true, maxSockets: 32 });
44+ const chunkSize = o.chunkSize ?? 64 * 1024 * 1024;
45+
46+ /** One upstream request with a fully buffered (or empty) body. */
47+ function send(method: string, path: string, headers: http.OutgoingHttpHeaders, body?: Buffer): Promise<{ status: number; headers: http.IncomingHttpHeaders; body: Buffer }> {
48+ return new Promise((resolve, reject) => {
49+ const req = client.request({ protocol: upstream.protocol, hostname: upstream.hostname, port: upstream.port || undefined, method, path, headers: { ...headers, host: upstream.host, "content-length": String(body?.length ?? 0) }, agent }, (res) => {
50+ const parts: Buffer[] = [];
51+ res.on("data", (d: Buffer) => parts.push(d));
52+ res.on("end", () => resolve({ status: res.statusCode ?? 502, headers: res.headers, body: Buffer.concat(parts) }));
53+ res.on("error", reject);
54+ });
55+ req.on("error", reject);
56+ req.end(body);
57+ });
58+ }
59+
60+ function forwardHeaders(req: http.IncomingMessage): http.OutgoingHttpHeaders {
61+ const h: http.OutgoingHttpHeaders = {};
62+ for (const [k, v] of Object.entries(req.headers)) if (!HOP_BY_HOP.has(k) && v !== undefined) h[k] = v;
63+ return h;
64+ }
65+
66+ /** Streams a request and its response straight through. */
67+ function passThrough(req: http.IncomingMessage, res: http.ServerResponse): void {
68+ const headers = forwardHeaders(req);
69+ if (req.headers["content-length"]) headers["content-length"] = req.headers["content-length"];
70+ const up = client.request({ protocol: upstream.protocol, hostname: upstream.hostname, port: upstream.port || undefined, method: req.method, path: req.url, headers: { ...headers, host: upstream.host }, agent }, (r) => {
71+ const out: http.OutgoingHttpHeaders = {};
72+ for (const [k, v] of Object.entries(r.headers)) if (!HOP_BY_HOP.has(k) || k === "content-length") out[k] = v;
73+ res.writeHead(r.statusCode ?? 502, out);
74+ r.pipe(res);
75+ });
76+ up.on("error", (e) => {
77+ log(`relay: ${req.method} ${req.url} failed: ${e.message}`);
78+ if (!res.headersSent) res.writeHead(502, { "content-type": "application/json" });
79+ res.end(JSON.stringify({ errors: [{ code: "UNKNOWN", message: `registry relay: ${e.message}` }] }));
80+ });
81+ req.pipe(up);
82+ }
83+
84+ /**
85+ * A blob upload (PATCH or PUT to an upload URL): sent on in chunks. Each full chunk
86+ * goes as a PATCH with Content-Range; a PUT's remainder goes with the final PUT.
87+ */
88+ async function chunkedUpload(req: http.IncomingMessage, res: http.ServerResponse): Promise<void> {
89+ const url = new URL(req.url!, "http://relay");
90+ const headers = forwardHeaders(req);
91+ delete headers["content-range"];
92+ delete headers["content-type"];
93+ let location = url.pathname;
94+ // Where the upload stands: earlier chunks may have come through another connection.
95+ const status = await send("GET", location, { authorization: req.headers.authorization ?? "" });
96+ if (status.status !== 204) return reply(res, status);
97+ let offset = uploadSize(status.headers);
98+ const declaredStart = /^(?:bytes[ =])?(\d+)-/.exec(String(req.headers["content-range"] ?? ""));
99+ if (declaredStart && Number(declaredStart[1]) !== offset) return reply(res, status, 416);
100+
101+ let buf: Buffer = Buffer.alloc(0);
102+ let last: Awaited<ReturnType<typeof send>> | null = null;
103+ const flush = async (data: Buffer) => {
104+ for (let attempt = 1; ; attempt++) {
105+ try {
106+ const r = await send("PATCH", location, { ...headers, "content-type": "application/octet-stream", "content-range": `${offset}-${offset + data.length - 1}` }, data);
107+ if (r.status === 202) {
108+ location = new URL(String(r.headers.location ?? location), "http://relay").pathname;
109+ offset += data.length;
110+ last = r;
111+ return true;
112+ }
113+ if (r.status < 500 || attempt >= 3) {
114+ last = r;
115+ return false;
116+ }
117+ } catch (e) {
118+ if (attempt >= 3) throw e;
119+ }
120+ // Retry from wherever the registry got to.
121+ const s = await send("GET", location, { authorization: req.headers.authorization ?? "" });
122+ if (s.status === 204 && uploadSize(s.headers) === offset + data.length) {
123+ offset += data.length;
124+ last = s;
125+ return true;
126+ }
127+ await new Promise((r) => setTimeout(r, 1000 * attempt));
128+ }
129+ };
130+ for await (const piece of req as AsyncIterable<Buffer>) {
131+ buf = buf.length ? Buffer.concat([buf, piece]) : piece;
132+ while (buf.length >= chunkSize) {
133+ const chunk = buf.subarray(0, chunkSize);
134+ buf = Buffer.from(buf.subarray(chunkSize));
135+ if (!(await flush(chunk))) return reply(res, last!);
136+ }
137+ }
138+ if (req.method === "PUT") {
139+ const r = await send("PUT", `${location}${url.search}`, { ...headers, ...(buf.length ? { "content-type": "application/octet-stream", "content-range": `${offset}-${offset + buf.length - 1}` } : {}) }, buf);
140+ return reply(res, r);
141+ }
142+ if (buf.length && !(await flush(buf))) return reply(res, last!);
143+ if (!last) last = await send("GET", location, { authorization: req.headers.authorization ?? "" });
144+ // The client sent one PATCH; answer it like the registry would, covering everything so far.
145+ return reply(res, { ...last, status: 202, headers: { ...last.headers, location, range: `0-${Math.max(0, offset - 1)}` } }, 202);
146+ }
147+
148+ function reply(res: http.ServerResponse, r: { status: number; headers: http.IncomingHttpHeaders; body: Buffer }, status = r.status): void {
149+ const out: http.OutgoingHttpHeaders = {};
150+ for (const [k, v] of Object.entries(r.headers)) if (!HOP_BY_HOP.has(k)) out[k] = v;
151+ out["content-length"] = String(r.body.length);
152+ res.writeHead(status, out);
153+ res.end(r.body);
154+ }
155+
156+ // Intercepted HTTPS traffic for the Diffinery host.
157+ const inner = http.createServer((req, res) => {
158+ const big = (req.method === "PATCH" || req.method === "PUT") && UPLOAD_PATH.test(req.url ?? "");
159+ if (!big) return passThrough(req, res);
160+ chunkedUpload(req, res).catch((e: Error) => {
161+ log(`relay: upload ${req.url} failed: ${e.message}`);
162+ if (!res.headersSent) res.writeHead(502, { "content-type": "application/json" });
163+ res.end(JSON.stringify({ errors: [{ code: "BLOB_UPLOAD_INVALID", message: `registry relay: ${e.message}` }] }));
164+ });
165+ });
166+ inner.requestTimeout = 0;
167+ inner.headersTimeout = 120_000;
168+ const secure = { cert: o.cert, key: o.key };
169+
170+ // The proxy itself: CONNECT tunnels, plus plain-HTTP proxying for completeness.
171+ const proxy = http.createServer((req, res) => {
172+ let target: URL;
173+ try {
174+ target = new URL(req.url ?? "");
175+ } catch {
176+ res.writeHead(400).end("absolute URL required");
177+ return;
178+ }
179+ const up = http.request(target, { method: req.method, headers: { ...forwardHeaders(req), host: target.host, ...(req.headers["content-length"] ? { "content-length": req.headers["content-length"] } : {}) } }, (r) => {
180+ res.writeHead(r.statusCode ?? 502, r.headers);
181+ r.pipe(res);
182+ });
183+ up.on("error", () => res.writeHead(502).end());
184+ req.pipe(up);
185+ });
186+ proxy.requestTimeout = 0;
187+ proxy.on("connect", (req: http.IncomingMessage, socket: net.Socket, head: Buffer) => {
188+ const [host, portStr] = (req.url ?? "").split(":");
189+ const port = Number(portStr || 443);
190+ socket.on("error", () => socket.destroy());
191+ if (host?.toLowerCase() === o.host.toLowerCase() && port === 443) {
192+ socket.write("HTTP/1.1 200 Connection Established\r\n\r\n");
193+ if (head.length) socket.unshift(head);
194+ const t = new tls.TLSSocket(socket, { isServer: true, ...secure, ALPNProtocols: ["http/1.1"] });
195+ t.on("error", () => t.destroy());
196+ inner.emit("connection", t);
197+ return;
198+ }
199+ const out = net.connect(port, host!, () => {
200+ socket.write("HTTP/1.1 200 Connection Established\r\n\r\n");
201+ if (head.length) out.write(head);
202+ out.pipe(socket);
203+ socket.pipe(out);
204+ });
205+ out.on("error", () => {
206+ if (socket.writable) socket.end("HTTP/1.1 502 Bad Gateway\r\n\r\n");
207+ socket.destroy();
208+ });
209+ });
210+
211+ await new Promise<void>((resolve, reject) => {
212+ proxy.once("error", reject);
213+ proxy.listen(o.port, "127.0.0.1", () => resolve());
214+ });
215+ const port = (proxy.address() as net.AddressInfo).port;
216+ log(`registry relay for ${o.host} -> ${o.upstream} listening on 127.0.0.1:${port}`);
217+ return {
218+ port,
219+ close: () =>
220+ new Promise<void>((resolve) => {
221+ agent.destroy();
222+ inner.close();
223+ proxy.closeAllConnections();
224+ proxy.close(() => resolve());
225+ }),
226+ };
227+}
228+
229+/** Bytes the registry holds for an upload. */
230+function uploadSize(h: http.IncomingHttpHeaders): number {
231+ const exact = Number(h["diffinery-upload-size"]);
232+ if (Number.isInteger(exact) && exact >= 0) return exact;
233+ const m = /^(?:bytes=)?(\d+)-(\d+)$/.exec(String(h.range ?? ""));
234+ return m && Number(m[2]) > 0 ? Number(m[2]) + 1 : 0;
235+}
addedcontainers/runner/test/relay.test.ts+173−0View file
@@ -1,0 +1,173 @@
1+import { execFileSync } from "node:child_process";
2+import { createHash } from "node:crypto";
3+import { readFileSync } from "node:fs";
4+import http from "node:http";
5+import net from "node:net";
6+import path from "node:path";
7+import tls from "node:tls";
8+import { afterEach, beforeEach, describe, expect, it } from "vitest";
9+import { startRelay, type Relay } from "../src/relay.js";
10+import { tmp } from "./helpers.js";
11+
12+/**
13+ * The registry relay against a fake registry: blob uploads arrive in chunks no bigger
14+ * than the limit, everything else passes through, and other hosts are tunnelled.
15+ */
16+
17+const HOST = "registry.test";
18+const sha = (b: Buffer) => `sha256:${createHash("sha256").update(b).digest("hex")}`;
19+
20+interface Upload {
21+ data: Buffer[];
22+ size: number;
23+}
24+
25+/** Just enough of the OCI upload API to see what the relay sends. */
26+function fakeRegistry() {
27+ const uploads = new Map<string, Upload>();
28+ const blobs = new Map<string, Buffer>();
29+ const requests: Array<{ method: string; url: string; length: number; range?: string }> = [];
30+ let next = 0;
31+ const server = http.createServer(async (req, res) => {
32+ const parts: Buffer[] = [];
33+ for await (const p of req) parts.push(p as Buffer);
34+ const body = Buffer.concat(parts);
35+ const url = new URL(req.url!, "http://x");
36+ requests.push({ method: req.method!, url: req.url!, length: body.length, range: req.headers["content-range"] as string | undefined });
37+ if (req.headers.authorization !== "Basic dTpw") return res.writeHead(401).end();
38+ const head = (u: Upload, id: string) => ({ location: `/v2/o/img/blobs/uploads/${id}`, range: `0-${Math.max(0, u.size - 1)}`, "diffinery-upload-size": String(u.size) });
39+ let m: RegExpExecArray | null;
40+ if (req.method === "POST" && url.pathname === "/v2/o/img/blobs/uploads/") {
41+ const id = `u${++next}`;
42+ uploads.set(id, { data: [], size: 0 });
43+ return res.writeHead(202, head(uploads.get(id)!, id)).end();
44+ }
45+ if ((m = /^\/v2\/o\/img\/blobs\/uploads\/(\w+)$/.exec(url.pathname))) {
46+ const u = uploads.get(m[1]!);
47+ if (!u) return res.writeHead(404).end();
48+ if (req.method === "GET") return res.writeHead(204, head(u, m[1]!)).end();
49+ const cr = req.headers["content-range"] as string | undefined;
50+ if (cr && Number(cr.split("-")[0]) !== u.size) return res.writeHead(416, head(u, m[1]!)).end();
51+ if (body.length > 1024 * 1024) return res.writeHead(413).end("too large");
52+ u.data.push(body);
53+ u.size += body.length;
54+ if (req.method === "PATCH") return res.writeHead(202, head(u, m[1]!)).end();
55+ const all = Buffer.concat(u.data);
56+ if (sha(all) !== url.searchParams.get("digest")) return res.writeHead(400).end("digest");
57+ blobs.set(sha(all), all);
58+ return res.writeHead(201, { location: `/v2/o/img/blobs/${sha(all)}`, "docker-content-digest": sha(all) }).end();
59+ }
60+ if ((m = /^\/v2\/o\/img\/blobs\/(sha256:\w+)$/.exec(url.pathname))) {
61+ const b = blobs.get(m[1]!);
62+ return b ? res.writeHead(200, { "content-length": String(b.length) }).end(b) : res.writeHead(404).end();
63+ }
64+ res.writeHead(404).end();
65+ });
66+ return { server, uploads, blobs, requests };
67+}
68+
69+let reg: ReturnType<typeof fakeRegistry>;
70+let relay: Relay;
71+let cert: string;
72+
73+beforeEach(async () => {
74+ const dir = tmp();
75+ execFileSync("openssl", ["req", "-x509", "-newkey", "ec", "-pkeyopt", "ec_paramgen_curve:prime256v1", "-nodes", "-days", "1", "-subj", `/CN=${HOST}`, "-addext", `subjectAltName=DNS:${HOST}`, "-keyout", path.join(dir, "k.pem"), "-out", path.join(dir, "c.pem")], { stdio: "ignore" });
76+ cert = readFileSync(path.join(dir, "c.pem"), "utf8");
77+ reg = fakeRegistry();
78+ await new Promise<void>((r) => reg.server.listen(0, "127.0.0.1", () => r()));
79+ const upstream = `http://127.0.0.1:${(reg.server.address() as net.AddressInfo).port}`;
80+ relay = await startRelay({ port: 0, host: HOST, upstream, cert, key: readFileSync(path.join(dir, "k.pem"), "utf8"), chunkSize: 1024 * 1024, log: () => {} });
81+});
82+
83+afterEach(async () => {
84+ await relay.close();
85+ reg.server.closeAllConnections();
86+ await new Promise((r) => reg.server.close(r));
87+});
88+
89+/** An HTTPS request to the registry host through the relay, as the Docker daemon makes it. */
90+function viaRelay(method: string, url: string, body?: Buffer, headers: Record<string, string> = {}): Promise<{ status: number; headers: http.IncomingHttpHeaders; body: Buffer }> {
91+ return new Promise((resolve, reject) => {
92+ const connect = http.request({ host: "127.0.0.1", port: relay.port, method: "CONNECT", path: `${HOST}:443` });
93+ connect.on("error", reject);
94+ connect.on("connect", (_res, socket) => {
95+ const secure = tls.connect({ socket, servername: HOST, ca: cert });
96+ const req = http.request({ method, path: url, headers: { host: HOST, authorization: "Basic dTpw", ...headers }, createConnection: () => secure }, (res) => {
97+ const parts: Buffer[] = [];
98+ res.on("data", (d: Buffer) => parts.push(d));
99+ res.on("end", () => {
100+ secure.destroy();
101+ resolve({ status: res.statusCode!, headers: res.headers, body: Buffer.concat(parts) });
102+ });
103+ });
104+ req.on("error", reject);
105+ // Stream the body in pieces, with no Content-Length, like docker does.
106+ if (body) for (let i = 0; i < body.length; i += 300_000) req.write(body.subarray(i, i + 300_000));
107+ req.end();
108+ });
109+ connect.end();
110+ });
111+}
112+
113+describe("registry relay", () => {
114+ it("splits a single-PUT upload (containerd) into PATCH chunks under the limit", async () => {
115+ const blob = Buffer.alloc(2_600_000, 7);
116+ const start = await viaRelay("POST", "/v2/o/img/blobs/uploads/");
117+ expect(start.status).toBe(202);
118+ const done = await viaRelay("PUT", `${start.headers.location}?digest=${sha(blob)}`, blob);
119+ expect(done.status).toBe(201);
120+ expect(done.headers["docker-content-digest"]).toBe(sha(blob));
121+ expect(reg.blobs.get(sha(blob))?.equals(blob)).toBe(true);
122+ const sent = reg.requests.filter((r) => r.method === "PATCH" || r.method === "PUT");
123+ expect(sent.map((r) => [r.method, r.length])).toEqual([
124+ ["PATCH", 1048576],
125+ ["PATCH", 1048576],
126+ ["PUT", 502848],
127+ ]);
128+ expect(sent[1]!.range).toBe("1048576-2097151");
129+ });
130+
131+ it("answers a streamed PATCH (docker) like the registry would, then completes with an empty PUT", async () => {
132+ const blob = Buffer.from(Array.from({ length: 2_300_000 }, (_, i) => i % 251));
133+ const start = await viaRelay("POST", "/v2/o/img/blobs/uploads/");
134+ const patch = await viaRelay("PATCH", String(start.headers.location), blob);
135+ expect(patch.status).toBe(202);
136+ expect(patch.headers.range).toBe(`0-${blob.length - 1}`);
137+ expect(reg.requests.filter((r) => r.method === "PATCH").length).toBe(3);
138+ const put = await viaRelay("PUT", `${patch.headers.location}?digest=${sha(blob)}`);
139+ expect(put.status).toBe(201);
140+ const back = await viaRelay("GET", `/v2/o/img/blobs/${sha(blob)}`);
141+ expect(back.status).toBe(200);
142+ expect(back.body.equals(blob)).toBe(true);
143+ });
144+
145+ it("passes the registry's errors back", async () => {
146+ const start = await viaRelay("POST", "/v2/o/img/blobs/uploads/");
147+ const bad = await viaRelay("PUT", `${start.headers.location}?digest=${sha(Buffer.from("other"))}`, Buffer.from("content"));
148+ expect(bad.status).toBe(400);
149+ expect((await viaRelay("GET", "/v2/o/img/blobs/uploads/nope")).status).toBe(404);
150+ expect((await viaRelay("GET", "/v2/o/img/blobs/uploads/u1", undefined, { authorization: "Basic bm86bm8=" })).status).toBe(401);
151+ });
152+
153+ it("tunnels other hosts untouched", async () => {
154+ const echo = net.createServer((s) => s.pipe(s));
155+ await new Promise<void>((r) => echo.listen(0, "127.0.0.1", () => r()));
156+ const port = (echo.address() as net.AddressInfo).port;
157+ const reply = await new Promise<string>((resolve, reject) => {
158+ const connect = http.request({ host: "127.0.0.1", port: relay.port, method: "CONNECT", path: `127.0.0.1:${port}` });
159+ connect.on("error", reject);
160+ connect.on("connect", (res, socket) => {
161+ expect(res.statusCode).toBe(200);
162+ socket.once("data", (d) => {
163+ resolve(d.toString());
164+ socket.destroy();
165+ });
166+ socket.write("ping");
167+ });
168+ connect.end();
169+ });
170+ expect(reply).toBe("ping");
171+ echo.close();
172+ });
173+});
addeddocs/container-registry.md+176−0View file
@@ -1,0 +1,176 @@
1+# Container registry
2+
3+Diffinery hosts container images at `diffinery.com`, next to the code that builds
4+them. It speaks the standard OCI distribution API, so `docker`, `podman`,
5+`buildx`, `crane`, `oras`, `skopeo` and the `docker/*` GitHub Actions work
6+without changes. Permissions follow ghcr.io: images belong to an owner, are
7+private by default, and can be linked to a repository whose collaborators then
8+share access to them.
9+
10+## Image names
11+
12+```
13+diffinery.com/<owner>/<name>[:<tag>|@sha256:<digest>]
14+```
15+
16+`<owner>` is a user or organization. `<name>` may contain slashes
17+(`diffinery.com/acme/api/worker`). Names are lowercase; `docker` requires that
18+anyway. Blobs and manifests are stored in R2, and identical layers are stored
19+once across all packages.
20+
21+## Signing in
22+
23+Use any username and a Diffinery token as the password. Create a token under
24+**Settings → Tokens**: the `read` scope pulls, `write` pushes, and `admin` also
25+deletes. A token limited to some repositories only reaches packages linked to
26+those repositories.
27+
28+```sh
29+echo "$DIFFINERY_TOKEN" | docker login diffinery.com -u me --password-stdin
30+docker push diffinery.com/acme/api:1.4.0
31+docker pull diffinery.com/acme/api:1.4.0
32+```
33+
34+## In workflows
35+
36+The job's `GITHUB_TOKEN` signs in, exactly as with ghcr.io. Its `packages`
37+permission decides what it can do:
38+
39+| `permissions` | `GITHUB_TOKEN` can |
40+|---|---|
41+| none set, or `write-all` | push and pull |
42+| `packages: write` | push and pull |
43+| `packages: read`, or `read-all` | pull |
44+| a map without `packages` | nothing |
45+
46+With write, a workflow pushes to packages linked to its own repository,
47+creates new packages under its repository's owner (linked to the repository),
48+and deletes versions of its repository's packages. With read, it pulls its
49+own repository's packages and any other package of the same owner that it can
50+see.
51+
52+```yaml
53+jobs:
54+ image:
55+ runs-on: ubuntu-latest
56+ permissions:
57+ contents: read
58+ packages: write
59+ steps:
60+ - uses: actions/checkout@v4
61+ - uses: docker/setup-buildx-action@v3
62+ - uses: docker/login-action@v3
63+ with:
64+ registry: diffinery.com
65+ username: ${{ github.actor }}
66+ password: ${{ secrets.GITHUB_TOKEN }}
67+ - uses: docker/build-push-action@v6
68+ with:
69+ push: true
70+ tags: diffinery.com/${{ github.repository }}:${{ github.sha }}
71+```
72+
73+`github.repository` keeps the owner's capitals; use a lowercase name if the
74+owner or repository has any (`${GITHUB_REPOSITORY,,}` in a shell step).
75+
76+Plain `docker login` / `docker build` / `docker push` in `run:` steps works the
77+same way. A later job pulls with `docker pull`, as a job `container:`, or as a
78+`services:` image, after signing in with `credentials:`:
79+
80+```yaml
81+ container:
82+ image: diffinery.com/acme/api:${{ github.sha }}
83+ credentials:
84+ username: ${{ github.actor }}
85+ password: ${{ secrets.GITHUB_TOKEN }}
86+```
87+
88+## Moving from ghcr.io
89+
90+Replace `ghcr.io` with `diffinery.com` in image names and in `registry:` for
91+`docker/login-action`. Nothing else changes: the same `permissions: packages:
92+write`, the same `GITHUB_TOKEN`, the same `org.opencontainers.image.source`
93+label. To copy existing images across, use a tool that copies all platforms
94+(this works for layers under 100 MB; for bigger ones, `docker pull`, `docker
95+tag` and `docker push` in a hosted workflow, see [Size limits](#size-limits)):
96+
97+```sh
98+crane auth login diffinery.com -u me -p "$DIFFINERY_TOKEN"
99+crane copy ghcr.io/acme/api:1.4.0 diffinery.com/acme/api:1.4.0
100+```
101+
102+## Repositories and visibility
103+
104+A package linked to a repository shares the repository's roles: readers pull,
105+writers push, admins delete versions and change settings. The owner (a user,
106+or an organization's owners and admins) always administers its packages, and
107+organization members can push to packages that aren't linked to a repository.
108+
109+A package is linked when it is created:
110+
111+- by a workflow, to the workflow's repository;
112+- by a person pushing `<owner>/<repo>` or `<owner>/<repo>/<anything>` while
113+ able to write to that repository;
114+- by the `org.opencontainers.image.source` label or annotation on the image,
115+ when it names a repository of the same owner (on Diffinery or on github.com)
116+ that the pusher can write to. An unlinked package is linked the first time
117+ an image with the label is pushed.
118+
119+The link can also be changed on the package's settings page or through the
120+API. Packages are **private** by default. **Internal** lets every signed-in
121+account pull; **public** lets anyone pull without signing in.
122+
123+Pages:
124+
125+- `https://diffinery.com/<owner>/-/packages`: an owner's packages.
126+- `https://diffinery.com/<owner>/-/packages/container/<name>`: tags, versions,
127+ pull commands and settings.
128+- `https://diffinery.com/<owner>/<repo>/packages`: the packages linked to a
129+ repository (the **Packages** tab).
130+
131+## API
132+
133+The REST API follows GitHub's packages API under `/api/v3`, so existing
134+scripts work after changing the base URL. `<type>` is always `container`;
135+`<name>` is URL-encoded when it contains slashes (`api%2Fworker`).
136+
137+| Request | Does |
138+|---|---|
139+| `GET /user/packages?package_type=container` | your packages |
140+| `GET /users/<user>/packages`, `GET /orgs/<org>/packages` | an owner's packages you can see (`?visibility=` filters) |
141+| `GET …/packages/<type>/<name>` | one package |
142+| `GET …/packages/<type>/<name>/versions` | its versions, newest first, with their tags in `metadata.container.tags` |
143+| `GET …/packages/<type>/<name>/versions/<id>` | one version |
144+| `DELETE …/packages/<type>/<name>/versions/<id>` | delete a version and its tags |
145+| `DELETE …/packages/<type>/<name>` | delete the package |
146+| `PATCH …/packages/<type>/<name>` | Diffinery only: `{"visibility": "public"}` or `{"repository": "api"}` (`null` unlinks) |
147+| `GET /repos/<owner>/<repo>/packages` | Diffinery only: packages linked to a repository |
148+
149+The registry itself is at `/v2/`: blob uploads (monolithic, chunked and
150+streamed), cross-repository mounts, range requests, image manifests and
151+indexes (OCI and Docker v2), tag listing with pagination, `DELETE` of
152+manifests and tags, the referrers API (signatures and SBOMs attached with
153+`subject`), and `/v2/_catalog` (the repositories you can pull).
154+
155+## Size limits
156+
157+Cloudflare limits one request to 100 MB, and `docker push` sends each layer
158+in a single request. On Diffinery's hosted runners this doesn't matter: the
159+runner's Docker daemon (and BuildKit builders created with
160+`docker buildx create` or `docker/setup-buildx-action`) reach the registry
161+through a local relay that splits uploads into smaller chunks. Layers of any
162+size push from workflows that use `docker` or BuildKit. Other clients run
163+directly in a step (`crane`, `skopeo`) don't go through the relay.
164+
165+From a laptop or a self-hosted runner, a layer over 100 MB fails to push with
166+`413` or a reset connection, because the common clients (`docker`, `buildx`,
167+`crane`, `skopeo`) send a layer in one request. The registry accepts chunked
168+uploads (`PATCH` with `Content-Range`) of any total size, so a client that
169+chunks works. Otherwise push from a hosted workflow, or keep layers under
170+100 MB by putting large dependencies in their own `RUN`/`COPY` steps. Pulls
171+have no size limit.
172+
173+## Cleanup
174+
175+Unfinished uploads are discarded after a day. Deleting a version removes its
176+tags; layers no other image uses are removed from storage by a scheduled job.