Diffinery

codician/diffinerypublic

Open-source, agent-first GitHub and GitHub Actions alternative that runs on your own Cloudflare account: git hosting, pull requests, code review and drop-in GitHub Actions YAML on Workers, Durable Objects, R2 and Containers.

Import repositories and organizations from GitHub (#36)

A server-side importer (web, REST and MCP) brings over git history with LFS,
issues and pull requests with their numbers, comments and reviews, labels,
milestones, releases, branch protection, Actions variables and deployment
environments. Secret names are listed for the owner to set by hand; values
are never read. Imports run as a durable workflow with paged, retried steps
in their own container, and imported history goes through push protection.

The sync's wait for Diffinery's Actions also gets a step timeout above the
script's own deadline.


Claude-Session: https://claude.ai/code/session_01AizKXSsengXZRegJxRATkL

Co-authored-by: Claude <noreply@anthropic.com>
Canberk Demirci authored · GitHub committed signed1 parent 4cfbb13commit ffe649f65f7b5b1c6ab1b39d7e40ff372584d05bBrowse files
49 files changed+6474−72
Files (49)
modified.github/workflows/sync-to-diffinery.yml+2−0View file
@@ -131,4 +131,6 @@
131131 - name: Wait for Actions on Diffinery
132132 # Pushes to main start CI (and, once Diffinery deploys, the deploy) there.
133133 if: github.ref == 'refs/heads/main'
134+ # A backstop above the script's own 75-minute deadline, in case a request hangs.
135+ timeout-minutes: 80
134136 run: scripts/diffinery-wait-runs.sh "$ORG/$NAME" "${{ steps.push.outputs.want }}" 75
modifiedREADME.md+102−2View file
@@ -207,8 +207,10 @@
207207 `create_branch`, `create_pull_request` (with an `intent` for reviewers),
208208 `review_pull_request`, `merge_pull_request` (policy-guarded), issues,
209209 `list_workflow_runs` and `get_job_logs`, `rerun_workflow`, releases
210-(`create_release`, `upload_release_asset`, `generate_release_notes`), and
211-`sandbox_create`/`sandbox_exec` for running builds and tests.
210+(`create_release`, `upload_release_asset`, `generate_release_notes`),
211+`sandbox_create`/`sandbox_exec` for running builds and tests, and
212+`start_github_import`/`get_import`/`list_imports` to move repositories in
213+from GitHub (see below).
212214
213215 Agent guardrails apply everywhere: git, REST, MCP and the web UI.
214216
@@ -479,6 +481,104 @@
479481 commit search. Search qualifiers for stars, forks and languages are accepted
480482 but ignored.
481483
484+## Moving from GitHub
485+
486+Diffinery imports a GitHub repository — or a whole organization — on the server:
487+it copies the Git history in a container and the rest through GitHub's REST API,
488+so you don't clone or push anything yourself and large repositories don't run
489+through your laptop.
490+
491+Start an import three ways:
492+
493+- **Web** — *Settings → Imports → New*, or the *Import from GitHub* link on *New
494+ repository*. Paste `owner/name` (or a GitHub URL), choose where it lands and
495+ whether it stays public or private, and watch each step. For an organization,
496+ pick the repositories from the list.
497+- **CLI** — `diffinery migrate github octo/app` imports one repository and
498+ follows it to completion; `diffinery migrate github-org octo --to my-org`
499+ imports an organization. Both take `--json`. `--local` runs the older
500+ client-side path instead (a mirror push plus a best-effort recreation of
501+ issues and open pull requests).
502+- **MCP / agents** — `start_github_import`, then `get_import` to poll and
503+ `list_imports` to review. An agent can import only into its owner's account or
504+ an organization the owner administers.
505+
506+A private repository, Git LFS, Actions variables and branch protection need a
507+GitHub token (`GH_TOKEN`/`GITHUB_TOKEN`, or the token field in the form) with
508+`repo` access. For GitHub Enterprise you must name the API base explicitly with
509+`GITHUB_API_URL` (or the API URL field in the form): when a token is supplied for
510+a non-github.com source, Diffinery requires a matching `github_api_url` on the
511+same host and attaches the token only there, so a host taken from the source can
512+never receive it. Server-side fetches only ever reach that one host, every
513+response body is size-bounded, redirects are not followed across hosts, and the
514+token goes to GitHub alone (never logged, never returned).
515+
516+**Limits.** Each account (and each organization) may run **one** organization
517+import and **three** single-repository imports at a time. A repository larger
518+than `IMPORT_MAX_REPO_KB` (default 10 GB) is refused up front. The Git transfer
519+has a **6-hour** ceiling; an import that is still transferring after that fails
520+cleanly. Draft imports left unstarted for seven days are cancelled and their
521+stored token is erased.
522+
523+**Agents.** An agent may import only into its owner's account, or an organization
524+the owner administers, and only when the owner holds at least **write** there.
525+Agents never read or set secrets and never manage **variables**: the repository,
526+organization and environment **variable** steps are skipped for an
527+agent-started import (and re-checked against a human owner when they do run). An
528+agent-started import also never bypasses push protection — if the history is not
529+clean, it stops.
530+
531+What comes across, keeping original numbers and attribution:
532+
533+- **Git** — every branch and tag, and LFS objects. Pull-request head commits are
534+ kept so merged and open PRs still point at real commits. Merge-queue refs are
535+ skipped.
536+- **Issues and pull requests** — with their **original numbers** (gaps and all),
537+ bodies, comments, reviews and review threads, labels, milestones, assignees,
538+ merged/closed state and who merged.
539+- **Labels, milestones, branch protection**, and **Actions variables** (repository
540+ and organization) and **deployment environments** (names, branch policies and
541+ variables).
542+- **Releases** and their assets, where the server's releases feature is enabled.
543+
544+What does **not** come across:
545+
546+- **Secret values.** GitHub never discloses them, and Diffinery never tries to.
547+ Repository and environment secret **names** are listed so you can re-enter the
548+ values by hand (`diffinery secret set <NAME> -R owner/repo`), which an admin
549+ does — never an agent.
550+- **People.** No accounts are created. A GitHub user is attributed to a Diffinery
551+ account only through a verified link, matched by GitHub's **numeric id** so a
552+ rename can't mis-map and keyed by the full API base so a github.com link and an
553+ Enterprise link stay separate. Everyone else is imported as a **mannequin**: a
554+ non-account placeholder (`github:<login>`, linking to the GitHub profile) that
555+ carries the history but can't sign in, be assigned work or cast a vote. Where a
556+ mannequin authors a squashed merge, the commit uses GitHub's `noreply` address.
557+
558+Link a GitHub account under *Settings → Linked accounts → GitHub* one of two ways:
559+
560+- **Gist proof (no token).** Diffinery shows a one-time code; you put it in a
561+ **public gist** and confirm your login. Diffinery reads your public profile and
562+ gists **unauthenticated** on the API base you named and links the account only
563+ when it finds the code. Nothing but your public data is read.
564+- **OAuth (optional).** When `GITHUB_OAUTH_CLIENT_ID` and
565+ `GITHUB_OAUTH_CLIENT_SECRET` are configured, a *Connect GitHub* button runs the
566+ OAuth flow with state bound to your account so a callback can't be replayed.
567+
568+A site admin can clear any link (audited). Imported reviews are kept as history
569+but **never count as approvals**, even when their author is a linked account — a
570+pull request still needs a fresh review on Diffinery to be approved.
571+
572+Old history that contains a credential is imported with an audited push-protection
573+bypass (the history is already public on GitHub); choose *stop* to fail instead.
574+
575+**Retrying.** Only the person who started an import may retry it, and a retry
576+needs the GitHub token again (it is never stored past the run). While an import
577+is in progress the target repository is **locked** against other writes, and a
578+retry refuses to touch a repository that already holds content the import did not
579+write — it will not adopt or overwrite an existing repository. An agent may not
580+retry once the repository has been created.
581+
482582 ## Developing
483583
484584 ```sh
modifiedapps/cli/src/commands/migrate.ts+223−17View file
@@ -2,7 +2,7 @@
22 import { tmpdir } from "node:os";
33 import path from "node:path";
44 import { ApiError, type Api } from "../api.js";
5-import { parseRepoArg, repoPath, str, type Command, type Ctx, type Group, type RepoRef } from "../cli.js";
5+import { parseRepoArg, repoPath, str, type Command, type Ctx, type Group, type RepoRef, type Values } from "../cli.js";
66 import { basicAuthHeader, git, gitCapture, withGitConfig } from "../git.js";
77 import { GitHub, GitHubError } from "../github.js";
88 import { CliError } from "../io.js";
@@ -101,20 +101,21 @@
101101 ctx.out(`Using existing repository ${to.owner}/${to.name}`);
102102 return { repo: existing.data, created: false };
103103 }
104- // Diffinery has no "internal": a GitHub internal repository becomes private.
105- const visibility = forcePrivate || gh.private || (gh.visibility && gh.visibility !== "public") ? "private" : "public";
104+ // Diffinery has only public and private; GitHub "internal" repositories become private.
105+ const visibility = !forcePrivate && (gh.visibility === "public" || (gh.visibility === undefined && !gh.private)) ? "public" : "private";
106106 const body = { name: to.name, description: gh.description ?? undefined, visibility };
107107 const repo = await api.post<RepoJson>(to.owner === me ? "/user/repos" : `/orgs/${encodeURIComponent(to.owner)}/repos`, body);
108108 ctx.out(`${ctx.c.green("✓")} Created ${repo.full_name} (${repo.visibility})`);
109109 return { repo, created: true };
110110 }
111111
112-async function migrateGit(ctx: Ctx, gh: GhRepo, ghToken: string | null, target: string, token: string | null, diffineryUrl: string, s: Summary): Promise<Set<string>> {
112+async function migrateGit(ctx: Ctx, gh: GhRepo, target: string, token: string | null, diffineryUrl: string, s: Summary): Promise<Set<string>> {
113113 const dir = await mkdtemp(path.join(tmpdir(), "diffinery-migrate-"));
114114 const mirror = path.join(dir, "repo.git");
115115 try {
116116 ctx.out(`Cloning ${gh.html_url} …`);
117- const cloneEnv = withGitConfig(ctx.io.env, ghToken && /^https?:\/\//.test(gh.clone_url) ? [basicAuthHeader(new URL(gh.clone_url).origin, "x-access-token", ghToken)] : []);
117+ const cloneToken = /^https?:\/\//.test(gh.clone_url) ? ghTokenFor(ctx, gh.clone_url) : null;
118+ const cloneEnv = withGitConfig(ctx.io.env, cloneToken ? [basicAuthHeader(new URL(gh.clone_url).origin, "x-access-token", cloneToken)] : []);
118119 const cl = await gitCapture(ctx.io, ["clone", "--mirror", "--quiet", gh.clone_url, mirror], { env: { ...cloneEnv, GIT_TERMINAL_PROMPT: "0" } });
119120 if (cl.code !== 0) throw new CliError(`git clone --mirror failed: ${cl.stderr.trim()}`);
120121 const refs = (await git(ctx.io, ["for-each-ref", "--format=%(refname)", "refs/heads", "refs/tags"], { cwd: mirror })).split("\n").filter(Boolean);
@@ -215,33 +216,175 @@
215216 }
216217 }
217218
219+// ---------------------------------------------------------------- the server importer
220+
221+/** The import resource returned by `/api/v3/diffinery/imports` (the fields the CLI shows). */
222+interface ImportJson {
223+ id: string;
224+ html_url: string;
225+ status: "draft" | "queued" | "running" | "succeeded" | "failed" | "cancelled";
226+ kind: "repository" | "organization";
227+ source: { full_name: string };
228+ target: { full_name: string | null; html_url: string | null };
229+ steps: Array<{ name: string; title: string; status: "pending" | "running" | "done" | "skipped" | "failed"; message: string | null; count: number | null }>;
230+ git: { phase: string | null; message: string | null; bytes: number; chunks: number } | null;
231+ summary: { secretNames?: string[]; environments?: { secretNames: string[] } } & Record<string, unknown>;
232+ warnings: string[];
233+ error: string | null;
234+ children?: Array<{ id: string; source: string; target: string; status: string; step: string | null; error: string | null }>;
235+}
236+
237+const DONE = new Set(["succeeded", "failed", "cancelled"]);
238+const POLL_MS = 1500;
239+
240+/** Poll an import to completion, printing each step and Git phase as it changes. */
241+async function pollImport(ctx: Ctx, api: Api, id: string): Promise<ImportJson> {
242+ const seenStep = new Map<string, string>();
243+ let seenGit = "";
244+ let seenChild = new Map<string, string>();
245+ for (let i = 0; ; i++) {
246+ const imp = await api.get<ImportJson>(`/diffinery/imports/${encodeURIComponent(id)}`);
247+ if (!ctx.json) {
248+ for (const s of imp.steps) {
249+ const prev = seenStep.get(s.name);
250+ if (prev === s.status) continue;
251+ seenStep.set(s.name, s.status);
252+ if (s.status === "running") ctx.out(ctx.c.dim(` … ${s.title}${s.message ? ` (${s.message})` : ""}`));
253+ else if (s.status === "done") ctx.out(` ${ctx.c.green("✓")} ${s.title}${s.count != null ? ` (${s.count})` : s.message ? ` (${s.message})` : ""}`);
254+ else if (s.status === "skipped") ctx.out(ctx.c.dim(` – ${s.title}${s.message ? ` (${s.message})` : ""}`));
255+ else if (s.status === "failed") ctx.out(` ${ctx.c.red("X")} ${s.title}${s.message ? `: ${s.message}` : ""}`);
256+ }
257+ if (imp.git && imp.git.phase && imp.git.phase !== seenGit) {
258+ seenGit = imp.git.phase;
259+ ctx.out(ctx.c.dim(` … git: ${imp.git.message ?? imp.git.phase}`));
260+ }
261+ for (const ch of imp.children ?? []) {
262+ const key = `${ch.status}:${ch.step ?? ""}`;
263+ if (seenChild.get(ch.id) === key) continue;
264+ seenChild.set(ch.id, key);
265+ const mark = ch.status === "succeeded" ? ctx.c.green("✓") : ch.status === "failed" ? ctx.c.red("X") : ch.status === "cancelled" ? ctx.c.dim("–") : ctx.c.dim("…");
266+ ctx.out(` ${mark} ${ch.source} → ${ch.target}${ch.status === "running" && ch.step ? ctx.c.dim(` (${ch.step})`) : ch.error ? ctx.c.red(` ${ch.error}`) : ""}`);
267+ }
268+ }
269+ if (DONE.has(imp.status)) return imp;
270+ await ctx.io.sleep(POLL_MS);
271+ }
272+}
273+
274+/** Secret names across the repository and its environments, with the command to set each. */
275+function secretLines(ctx: Ctx, imp: ImportJson, target: string): void {
276+ const names = [...(imp.summary.secretNames ?? []), ...(imp.summary.environments?.secretNames ?? [])];
277+ if (!names.length) return;
278+ ctx.out("");
279+ ctx.out(`GitHub does not reveal secret values. Set these ${names.length} secret${names.length === 1 ? "" : "s"} by hand:`);
280+ for (const n of names) ctx.out(` diffinery secret set ${n} -R ${target}`);
281+}
282+
283+function renderImport(ctx: Ctx, imp: ImportJson): number {
284+ if (ctx.json) {
285+ ctx.out(JSON.stringify(imp, null, 2));
286+ return imp.status === "succeeded" ? 0 : 1;
287+ }
288+ const c = ctx.c;
289+ const target = imp.target.full_name ?? "—";
290+ ctx.out("");
291+ if (imp.status === "succeeded") ctx.out(c.bold(`Imported ${imp.source.full_name} → ${target}`));
292+ else if (imp.status === "cancelled") ctx.out(c.yellow(`Import of ${imp.source.full_name} was cancelled`));
293+ else ctx.out(c.red(`Import of ${imp.source.full_name} failed${imp.error ? `: ${imp.error}` : ""}`));
294+ for (const w of imp.warnings) ctx.err(`${c.yellow("warning:")} ${w}`);
295+ if (imp.status === "succeeded") {
296+ secretLines(ctx, imp, target);
297+ if (imp.target.html_url) {
298+ ctx.out("");
299+ ctx.out(imp.target.html_url);
300+ }
301+ }
302+ return imp.status === "succeeded" ? 0 : 1;
303+}
304+
305+const ghToken = (ctx: Ctx) => ctx.io.env.GH_TOKEN || ctx.io.env.GITHUB_TOKEN || null;
306+const ghApiUrl = (ctx: Ctx) => ctx.io.env.GITHUB_API_URL || undefined;
307+
308+/**
309+ * The GitHub token to send to a given URL, or null. M3: only attach it to github.com (its API/host)
310+ * or to the host of an explicitly configured GITHUB_API_URL — never to a host taken from the source
311+ * repository argument, so a crafted source cannot capture the token.
312+ */
313+function ghTokenFor(ctx: Ctx, url: string): string | null {
314+ const token = ghToken(ctx);
315+ if (!token) return null;
316+ let host: string;
317+ try {
318+ host = new URL(url).host.toLowerCase();
319+ } catch {
320+ return null;
321+ }
322+ if (host === "github.com" || host === "www.github.com" || host === "api.github.com") return token;
323+ const apiUrl = ctx.io.env.GITHUB_API_URL;
324+ if (apiUrl) {
325+ try {
326+ if (new URL(apiUrl).host.toLowerCase() === host) return token;
327+ } catch {
328+ // ignore a malformed GITHUB_API_URL
329+ }
330+ }
331+ return null;
332+}
333+
334+/** The server-side single-repository import: POST the request, then follow it to completion. */
335+async function runServer(ctx: Ctx, o: Values, srcArg: string): Promise<number> {
336+ const src = parseRepoArg(srcArg.replace(/^https?:\/\/github\.com\//, ""));
337+ const srcName = `${src.owner}/${src.name}`;
338+ const api = await ctx.authedApi();
339+ const to = str(o, "to") ? parseRepoArg(str(o, "to")!) : null;
340+ let owner = to?.owner;
341+ if (!owner) owner = (await api.get<{ login: string }>("/user")).login;
342+ const body = {
343+ source: srcName,
344+ owner,
345+ name: to?.name,
346+ visibility: o.private === true ? "private" : "keep",
347+ github_token: ghToken(ctx) ?? undefined,
348+ github_api_url: ghApiUrl(ctx),
349+ };
350+ if (!ctx.json) ctx.out(`Starting server import of ${srcName} …`);
351+ const created = await api.post<ImportJson>("/diffinery/imports", body);
352+ const final = await pollImport(ctx, api, created.id);
353+ return renderImport(ctx, final);
354+}
355+
218356 const github: Command = {
219357 name: "migrate github",
220- summary: "Import a GitHub repository (git data, and optionally issues)",
358+ summary: "Import a GitHub repository (git data, issues, pull requests, releases and more)",
221359 args: "<GH-OWNER/GH-REPO>",
222- description: `Creates the Diffinery repository if needed, mirrors all branches and tags (not refs/pull),
223-sets the default branch, copies Actions variables and lists Actions secret names to set.
224-With --issues, also copies labels, issues and comments (with attribution lines), and recreates
225-open pull requests whose branches were imported.
360+ description: `Runs a full server-side import: git data (keeping issue and pull-request numbers), labels,
361+milestones, issues and pull requests with comments and reviews, branch protection, Actions
362+variables, release assets and deployment environments. The command follows progress until the
363+import finishes. Secret values are never read from GitHub; their names are listed to set by hand.
226364
365+With --local, runs the old client-side import instead (mirror push plus a best-effort recreation
366+of issues and open pull requests); --issues then also copies labels, issues and comments.
367+
227368 GitHub access uses GH_TOKEN or GITHUB_TOKEN (needed for private repositories; GITHUB_API_URL
228369 selects a GitHub Enterprise API).`,
229370 options: {
230- to: { type: "string", short: "t", desc: "Diffinery repository to create or push into (default: <you>/<gh-repo>)", arg: "OWNER/NAME" },
231- issues: { type: "boolean", short: "i", desc: "Also import labels, issues, comments and open pull requests" },
371+ to: { type: "string", short: "t", desc: "Diffinery repository to create (default: <you>/<gh-repo>)", arg: "OWNER/NAME" },
232372 private: { type: "boolean", short: "p", desc: "Create the repository as private regardless of GitHub's visibility" },
373+ local: { type: "boolean", short: "l", desc: "Use the old client-side import (no server importer)" },
374+ issues: { type: "boolean", short: "i", desc: "With --local: also import labels, issues, comments and open pull requests" },
233375 },
234376 json: true,
235377 minArgs: 1,
236378 maxArgs: 1,
237379 async run(ctx, o, [srcArg]) {
380+ if (o.local !== true) return runServer(ctx, o, srcArg!);
238381 const src = parseRepoArg(srcArg!.replace(/^https?:\/\/github\.com\//, ""));
239382 const srcName = `${src.owner}/${src.name}`;
240383 const api = await ctx.authedApi();
241384 const cfg = await ctx.config();
242- const ghToken = ctx.io.env.GH_TOKEN || ctx.io.env.GITHUB_TOKEN || null;
243- const gh = new GitHub({ baseUrl: ctx.io.env.GITHUB_API_URL, token: ghToken, sleep: (ms) => ctx.io.sleep(ms), log: (m) => ctx.err(ctx.c.yellow(m)) });
244- if (!ghToken) ctx.err(ctx.c.yellow("note: GH_TOKEN/GITHUB_TOKEN is not set; only public data can be read and GitHub rate limits are low"));
385+ const token = ghToken(ctx);
386+ const gh = new GitHub({ baseUrl: ctx.io.env.GITHUB_API_URL, token, sleep: (ms) => ctx.io.sleep(ms), log: (m) => ctx.err(ctx.c.yellow(m)) });
387+ if (!token) ctx.err(ctx.c.yellow("note: GH_TOKEN/GITHUB_TOKEN is not set; only public data can be read and GitHub rate limits are low"));
245388
246389 const me = await api.get<{ login: string }>("/user");
247390 const gr = await gh.get<GhRepo>(`/repos/${srcName}`);
@@ -251,7 +394,7 @@
251394
252395 const { created } = await ensureRepo(ctx, api, to, me.login, gr, o.private === true);
253396 s.created = created;
254- const branches = await migrateGit(ctx, gr, ghToken, target, cfg.token, cfg.url, s);
397+ const branches = await migrateGit(ctx, gr, target, cfg.token, cfg.url, s);
255398 if (branches.has(gr.default_branch)) {
256399 await api.patch(repoPath(to), { default_branch: gr.default_branch });
257400 s.defaultBranch = gr.default_branch;
@@ -294,4 +437,67 @@
294437 },
295438 };
296439
297-export const migrateGroup: Group = { name: "migrate", summary: "Import repositories from other forges", commands: [github] };
440+const githubOrg: Command = {
441+ name: "migrate github-org",
442+ summary: "Import many repositories from a GitHub organization or user",
443+ args: "<GH-ORG-OR-USER>",
444+ description: `Starts a server-side bulk import of every repository in a GitHub organization or user account
445+(or just the ones named with --repos), into a Diffinery organization or user. Each repository is
446+imported in full, the same as "migrate github". The command follows progress until every
447+repository has finished.
448+
449+GitHub access uses GH_TOKEN or GITHUB_TOKEN; GITHUB_API_URL selects a GitHub Enterprise API.`,
450+ options: {
451+ to: { type: "string", short: "t", desc: "Diffinery owner to import into (default: you)", arg: "OWNER" },
452+ repos: { type: "string", desc: "Comma-separated repository names to import (default: all)", arg: "A,B,C" },
453+ private: { type: "boolean", short: "p", desc: "Create every repository as private" },
454+ "exclude-forks": { type: "boolean", desc: "Skip forks" },
455+ "exclude-archived": { type: "boolean", desc: "Skip archived repositories" },
456+ concurrency: { type: "string", short: "c", desc: "Repositories to import at once, 1–8 (default 3)", arg: "N" },
457+ },
458+ json: true,
459+ minArgs: 1,
460+ maxArgs: 1,
461+ async run(ctx, o, [orgArg]) {
462+ const org = orgArg!.replace(/^https?:\/\/github\.com\//, "").replace(/\/+$/, "");
463+ const api = await ctx.authedApi();
464+ const owner = str(o, "to") || (await api.get<{ login: string }>("/user")).login;
465+ const repositories = str(o, "repos")
466+ ? str(o, "repos")!
467+ .split(",")
468+ .map((s) => s.trim())
469+ .filter(Boolean)
470+ : undefined;
471+ const concurrency = str(o, "concurrency") ? Number(str(o, "concurrency")) : undefined;
472+ const body = {
473+ source_owner: org,
474+ owner,
475+ repositories,
476+ exclude_forks: o["exclude-forks"] === true || undefined,
477+ exclude_archived: o["exclude-archived"] === true || undefined,
478+ concurrency,
479+ visibility: o.private === true ? "private" : "keep",
480+ github_token: ghToken(ctx) ?? undefined,
481+ github_api_url: ghApiUrl(ctx),
482+ };
483+ if (!ctx.json) ctx.out(`Starting bulk import of ${org} into ${owner} …`);
484+ const created = await api.post<ImportJson>("/diffinery/imports", body);
485+ const final = await pollImport(ctx, api, created.id);
486+ if (ctx.json) {
487+ ctx.out(JSON.stringify(final, null, 2));
488+ return final.status === "succeeded" ? 0 : 1;
489+ }
490+ const c = ctx.c;
491+ const kids = final.children ?? [];
492+ const ok = kids.filter((k) => k.status === "succeeded").length;
493+ const failed = kids.filter((k) => k.status === "failed").length;
494+ ctx.out("");
495+ ctx.out(c.bold(`Bulk import of ${org}: ${ok}/${kids.length} repositories imported${failed ? c.red(`, ${failed} failed`) : ""}`));
496+ for (const w of final.warnings) ctx.err(`${c.yellow("warning:")} ${w}`);
497+ ctx.out("");
498+ ctx.out(final.html_url);
499+ return final.status === "succeeded" ? 0 : 1;
500+ },
501+};
502+
503+export const migrateGroup: Group = { name: "migrate", summary: "Import repositories from other forges", commands: [github, githubOrg] };
modifiedapps/cli/test/migrate.test.ts+44−3View file
@@ -67,7 +67,7 @@
6767 afterEach(() => srv.close());
6868
6969 it("creates the repo, pushes branches and tags, and copies issues, PRs, variables and secret names", async () => {
70- const r = await run(["migrate", "github", "gh-org/widgets", "--issues"], { env });
70+ const r = await run(["migrate", "github", "gh-org/widgets", "--local", "--issues"], { env });
7171 expect(r.stderr).not.toContain("error");
7272 expect(r.code).toBe(0);
7373
@@ -106,7 +106,7 @@
106106 });
107107
108108 it("repo clone authenticates for that command only", async () => {
109- await run(["migrate", "github", "gh-org/widgets"], { env });
109+ await run(["migrate", "github", "gh-org/widgets", "--local"], { env });
110110 const cwd = tempDir("diffinery-clone-");
111111 const r = await run(["repo", "clone", "alice/widgets", "w"], { env, cwd });
112112 expect(r.code).toBe(0);
@@ -120,13 +120,54 @@
120120 expect(anon.code).not.toBe(0);
121121 });
122122
123+ it("without --local, starts a server import and follows it to completion", async () => {
124+ const base = {
125+ id: "imp1",
126+ html_url: `${srv.url}/imports/imp1`,
127+ kind: "repository",
128+ source: { full_name: "gh-org/widgets" },
129+ target: { full_name: "alice/widgets", html_url: `${srv.url}/alice/widgets` },
130+ git: { phase: "done", message: "pushed", bytes: 123, chunks: 1 },
131+ summary: { secretNames: ["NPM_TOKEN"], environments: { secretNames: ["prod/DEPLOY_KEY"] } },
132+ warnings: [] as string[],
133+ error: null,
134+ children: [] as unknown[],
135+ };
136+ const steps = (s: string) => [
137+ { name: "repository", title: "Repository", status: "done", message: "alice/widgets", count: null },
138+ { name: "git", title: "Git data", status: s === "succeeded" ? "done" : "running", message: null, count: null },
139+ ];
140+ srv.on("POST", "/api/v3/diffinery/imports", { status: 201, body: { ...base, status: "queued", steps: steps("queued") } });
141+ let polls = 0;
142+ srv.on("GET", "/api/v3/diffinery/imports/imp1", () => {
143+ polls++;
144+ const status = polls >= 2 ? "succeeded" : "running";
145+ return { status: 200, body: { ...base, status, steps: steps(status) } };
146+ });
147+
148+ const r = await run(["migrate", "github", "gh-org/widgets"], { env });
149+ expect(r.code).toBe(0);
150+ // The request carried the source, resolved owner, keep-visibility and the GitHub token.
151+ const post = srv.find("POST", "/api/v3/diffinery/imports")[0]!.body as Record<string, unknown>;
152+ expect(post).toMatchObject({ source: "gh-org/widgets", owner: "alice", visibility: "keep", github_token: "ght" });
153+ // It polled until the import finished, and reported the result and the secrets to set by hand.
154+ expect(polls).toBeGreaterThanOrEqual(2);
155+ expect(r.stdout).toContain("Imported gh-org/widgets → alice/widgets");
156+ expect(r.stdout).toContain("diffinery secret set NPM_TOKEN -R alice/widgets");
157+ expect(r.stdout).toContain("diffinery secret set prod/DEPLOY_KEY -R alice/widgets");
158+ // The git data was NOT cloned or pushed by the CLI; the server does that.
159+ expect(srv.find("POST", "/api/v3/user/repos")).toHaveLength(0);
160+ // The GitHub token never went to Diffinery's own endpoints.
161+ for (const q of srv.requests) if (q.path.startsWith("/api/")) expect(q.headers.authorization).toBe("Bearer dfy_t");
162+ });
163+
123164 it("--private and --to override the target", async () => {
124165 srv.on("GET", "/api/v3/repos/team/w2", { status: 404, body: { message: "repository not found" } });
125166 srv.on("POST", "/api/v3/orgs/team/repos", { status: 201, body: { full_name: "team/w2", visibility: "private" } });
126167 srv.on("PATCH", "/api/v3/repos/team/w2", { body: {} });
127168 mkdirSync(path.join(gitRoot, "team"));
128169 execFileSync("git", ["init", "-q", "--bare", "w2.git"], { cwd: path.join(gitRoot, "team"), env });
129- const r = await run(["migrate", "github", "https://github.com/gh-org/widgets", "--to", "team/w2", "--private"], { env });
170+ const r = await run(["migrate", "github", "https://github.com/gh-org/widgets", "--local", "--to", "team/w2", "--private"], { env });
130171 expect(r.code).toBe(0);
131172 expect(srv.find("POST", "/api/v3/orgs/team/repos")[0]!.body).toMatchObject({ name: "w2", visibility: "private" });
132173 expect(srv.find("POST", /\/issues$/)).toHaveLength(0);
addedapps/diffinery/migrations/0010_imports.sql+56−0View file
@@ -1,0 +1,56 @@
1+-- Importing repositories from GitHub (one repository, or many from an organization or user).
2+
3+CREATE TABLE imports (
4+ id TEXT PRIMARY KEY,
5+ parent_id TEXT REFERENCES imports(id), -- bulk imports fan out into one child per repository
6+ kind TEXT NOT NULL CHECK (kind IN ('repo', 'bulk')),
7+ status TEXT NOT NULL CHECK (status IN ('draft', 'queued', 'running', 'succeeded', 'failed', 'cancelled')),
8+ created_by TEXT NOT NULL REFERENCES accounts(id),
9+ api_url TEXT NOT NULL, -- GitHub REST API base, e.g. https://api.github.com
10+ source TEXT NOT NULL, -- owner/name, or the owner for bulk imports
11+ target_owner_id TEXT NOT NULL REFERENCES accounts(id),
12+ target_name TEXT, -- repository imports only
13+ repo_id TEXT, -- the repository this import created
14+ request TEXT NOT NULL, -- JSON: visibility, options, selection
15+ steps TEXT NOT NULL DEFAULT '{}', -- JSON: per-step status and progress (written by the workflow)
16+ git TEXT, -- JSON: the Git transfer's progress and result (written by the agent)
17+ summary TEXT NOT NULL DEFAULT '{}', -- JSON: counts, secret names, findings
18+ warnings TEXT NOT NULL DEFAULT '[]',
19+ error TEXT,
20+ token_ciphertext TEXT, -- the GitHub token, encrypted; deleted when the import finishes
21+ push_token_id TEXT, -- the short-lived Diffinery token used for the Git transfer
22+ attempt INTEGER NOT NULL DEFAULT 1,
23+ created_at TEXT NOT NULL,
24+ updated_at TEXT NOT NULL,
25+ started_at TEXT,
26+ finished_at TEXT
27+);
28+CREATE INDEX imports_creator ON imports(created_by, created_at);
29+CREATE INDEX imports_parent ON imports(parent_id);
30+
31+-- Temporary tokens minted for an import's Git transfer carry its id.
32+ALTER TABLE tokens ADD COLUMN import_id TEXT;
33+
34+-- Verified links between Diffinery accounts and GitHub accounts, used to attribute imported work.
35+CREATE TABLE account_links (
36+ id TEXT PRIMARY KEY,
37+ account_id TEXT NOT NULL REFERENCES accounts(id),
38+ provider TEXT NOT NULL CHECK (provider IN ('github')),
39+ api_base TEXT NOT NULL, -- full normalised REST base, e.g. https://api.github.com or https://ghe.example.com/api/v3
40+ external_id TEXT NOT NULL,
41+ external_login TEXT NOT NULL,
42+ created_at TEXT NOT NULL,
43+ UNIQUE (provider, api_base, external_id)
44+);
45+CREATE INDEX account_links_account ON account_links(account_id);
46+
47+-- Actions variables shared by an organization's repositories.
48+CREATE TABLE org_variables (
49+ org_id TEXT NOT NULL REFERENCES accounts(id),
50+ name TEXT NOT NULL,
51+ value TEXT NOT NULL,
52+ updated_at TEXT NOT NULL,
53+ PRIMARY KEY (org_id, name)
54+);
55+
56+ALTER TABLE repos ADD COLUMN homepage TEXT NOT NULL DEFAULT '';
modifiedapps/diffinery/src/app.ts+6−1View file
@@ -8,7 +8,7 @@
88 import { getFirst } from "./lib/storage-keys.js";
99 import { BOOKMARK_COOKIE, BOOKMARK_HEADER, d1Session, sessionConstraint } from "./lib/d1.js";
1010 import { SESSION_COOKIE } from "./auth/sessions.js";
11-import { newId } from "./lib/util.js";
11+import { newId, now } from "./lib/util.js";
1212 import { deliverWebhook, type WebhookDeps, type WebhookRetry } from "./lib/webhooks.js";
1313 import { runStages, type Stage, type StageRetry } from "./lib/stages.js";
1414 import { apiNotFound, errorResponse, loadRepo, repoStub } from "./http/common.js";
@@ -227,5 +227,10 @@
227227 }),
228228 // Token expiry reminders and email log pruning.
229229 mailCron(env, new Directory(env), end).catch((e) => console.error("mail cron failed", e)),
230+ // Expire abandoned draft bulk imports after 7 days, and null their stored GitHub token (Low 5).
231+ env.DB.prepare("UPDATE imports SET status = 'cancelled', token_ciphertext = NULL, error = 'draft expired', finished_at = ?, updated_at = ? WHERE status = 'draft' AND created_at < ?")
232+ .bind(now(), now(), new Date(Date.now() - 7 * 86400_000).toISOString())
233+ .run()
234+ .catch((e) => console.error("draft import expiry failed", e)),
230235 ]);
231236 }
modifiedapps/diffinery/src/auth/principal.ts+12−2View file
@@ -7,7 +7,7 @@
77 import { ROLE_LEVEL } from "../repo/types.js";
88
99 export type Principal =
10- | { type: "account"; account: Account; via: "session" | "token" | "dev"; scopes: TokenScope[]; repoIds: string[] | null; sessionId?: string }
10+ | { type: "account"; account: Account; via: "session" | "token" | "dev"; scopes: TokenScope[]; repoIds: string[] | null; sessionId?: string; importId?: string; tokenId?: string }
1111 | { type: "repo-token"; repoId: string; token: string }
1212 | { type: "anonymous" };
1313
@@ -32,7 +32,8 @@
3232 if (/^basic$/i.test(scheme!)) {
3333 let bytes: Uint8Array;
3434 try {
35- bytes = fromB64(value.trim());
35+ // Git LFS may send the same header twice (http.extraHeader and the batch action's header), joined by a comma.
36+ bytes = fromB64(value.split(",")[0]!.trim());
3637 } catch {
3738 throw new HttpError(401, "malformed basic credentials");
3839 }
@@ -54,12 +55,21 @@
5455 return m ? m[1]! : null;
5556 }
5657
58+const IMPORT_GIT_PATH = /^\/[^/]+\/[^/]+\/(info\/refs|git-upload-pack|git-receive-pack|info\/lfs\/objects\/[^/]+)$/;
59+
5760 export async function resolvePrincipal(req: Request, env: Env, dir: Directory): Promise<Principal> {
5861 const cred = credentialFrom(req);
5962 if (cred) {
6063 if (cred.startsWith("dfy_")) {
6164 const v = await dir.verifyToken(cred);
6265 if (!v) throw new HttpError(401, "invalid, expired or revoked token");
66+ if (v.importId) {
67+ // An import's push token reaches only Git, Git LFS and its own agent endpoints.
68+ const path = new URL(req.url).pathname;
69+ const agent = path.startsWith(`/api/v3/diffinery/imports/${v.importId}/agent/`);
70+ if (!agent && !IMPORT_GIT_PATH.test(path)) throw new HttpError(403, "this token can only be used by its GitHub import");
71+ return { type: "account", account: v.account, via: "token", scopes: v.scopes, repoIds: v.repoIds, importId: v.importId, tokenId: v.id };
72+ }
6373 return { type: "account", account: v.account, via: "token", scopes: v.scopes, repoIds: v.repoIds };
6474 }
6575 if (cred.startsWith("djt_") || cred.startsWith("dst_")) {
modifiedapps/diffinery/src/env.ts+11−0View file
@@ -1,5 +1,6 @@
11 import type { DiffineryEvent } from "@diffinery/shared";
22 import type { RepoDO } from "./repo/RepoDO.js";
3+import type { ImportContainer } from "./imports/container.js";
34 import type { RunnerContainer, SandboxContainer } from "./runner/containers.js";
45 import type { MailJob } from "./mail/types.js";
56
@@ -15,6 +16,16 @@
1516 EMAIL?: SendEmail;
1617 /** Email jobs, sent by the queue consumer in mail/send.ts. */
1718 MAIL?: Queue<MailJob>;
19+ IMPORT_WORKFLOW: Workflow<{ importId: string }>;
20+ /** GitHub imports copy Git data in this container (absent under `wrangler dev`). */
21+ IMPORTER?: DurableObjectNamespace<ImportContainer>;
22+ /** "external": an import agent started outside the Worker claims the Git transfer (local development and tests). */
23+ IMPORT_GIT_AGENT?: string;
24+ /** Largest source repository an import will take, in KB (GitHub `size`); default 10 GB. */
25+ IMPORT_MAX_REPO_KB?: string;
26+ /** Optional GitHub OAuth app, for linking an account through the web flow instead of a gist. */
27+ GITHUB_OAUTH_CLIENT_ID?: string;
28+ GITHUB_OAUTH_CLIENT_SECRET?: string;
1829
1930 /** Canonical origin, e.g. https://diffinery.com */
2031 PUBLIC_URL: string;
modifiedapps/diffinery/src/http/api.ts+6−1View file
@@ -10,6 +10,7 @@
1010 import type { JobInfo, RunInfo } from "../repo/actions.js";
1111 import type { CommitInfo } from "../repo/RepoDO.js";
1212 import type { QueueEntry } from "../repo/queue.js";
13+import { imports } from "./imports.js";
1314 import { auditRepo, loadRepo, refuseAgent, repoStub } from "./common.js";
1415 import { repoCreated } from "../mail/notify.js";
1516 import { actionsRest } from "./actions-runtime.js";
@@ -51,6 +52,9 @@
5152 return n;
5253 };
5354
55+// GitHub imports, milestones, organization variables and linked accounts (mounted first: it extends PATCH /issues/:number).
56+api.route("/", imports);
57+
5458 // ---------------------------------------------------------------- meta & users
5559
5660 api.get("/", (c) => c.json({ name: "Diffinery", api: "v3", docs: `${c.env.PUBLIC_URL}/docs/api`, mcp: `${c.env.PUBLIC_URL}/mcp` }));
@@ -739,7 +743,8 @@
739743 });
740744 api.get("/repos/:owner/:repo/actions/variables", async (c) => {
741745 const { stub } = await loadRepo(c, P(c).owner, P(c).repo, "write");
742- const all = Object.entries(await stub.variables()).map(([name, value]) => ({ name, value }));
746+ // The repository's own variables; organization variables are listed under /orgs/:org/actions/variables.
747+ const all = Object.entries(await stub.variables(false)).map(([name, value]) => ({ name, value }));
743748 return c.json({ total_count: all.length, variables: paginate(c, all) });
744749 });
745750 api.delete("/repos/:owner/:repo/actions/variables/:name", async (c) => {
modifiedapps/diffinery/src/http/gh.ts+28−3View file
@@ -52,6 +52,23 @@
5252
5353 export function ghUser(base: string, u: Who | null | undefined) {
5454 if (!u) return null;
55+ // People imported from GitHub without a Diffinery account are mannequins: login "github:<login>",
56+ // type "Mannequin", pointing at their GitHub profile. They are clearly not Diffinery accounts.
57+ if (u.kind === "github" || u.login.startsWith("github:")) {
58+ const login = u.login.replace(/^github:/, "");
59+ return {
60+ login: `github:${login}`,
61+ id: numericId(u.id),
62+ node_id: nodeId("Mannequin", u.id),
63+ avatar_url: `${base}/api/avatars/${encodeURIComponent(login)}`,
64+ gravatar_id: "",
65+ url: `https://github.com/${encodeURIComponent(login)}`,
66+ html_url: `https://github.com/${encodeURIComponent(login)}`,
67+ type: "Mannequin",
68+ user_view_type: "public",
69+ site_admin: false,
70+ };
71+ }
5572 const api = `${apiRoot(base)}/users/${encodeURIComponent(u.login)}`;
5673 return {
5774 login: u.login,
@@ -84,6 +101,7 @@
84101 updated_at: string;
85102 pushed_at: string | null;
86103 topics?: string | null;
104+ homepage?: string | null;
87105 }
88106
89107 export function topicsOf(r: Pick<RepoRow, "topics">): string[] {
@@ -135,7 +153,7 @@
135153 ssh_url: `${html}.git`,
136154 clone_url: `${html}.git`,
137155 svn_url: html,
138- homepage: null,
156+ homepage: r.homepage || null,
139157 size: 0,
140158 stargazers_count: 0,
141159 watchers_count: 0,
@@ -216,6 +234,7 @@
216234 comments: number;
217235 locked?: boolean;
218236 lockReason?: string | null;
237+ milestone?: { number: number; title: string; state: "open" | "closed"; dueOn: string | null } | null;
219238 pull?: {
220239 head: string;
221240 headSha: string;
@@ -243,6 +262,12 @@
243262 }
244263
245264 const personByLogin = (x: IssueCtx, login: string): Who => x.people?.get(login.toLowerCase()) ?? { id: login, login, kind: "user" };
265+
266+/** A GitHub-shaped milestone object for embedding in an issue or pull request. */
267+function ghIssueMilestone(x: IssueCtx, m: { number: number; title: string; state: "open" | "closed"; dueOn: string | null }) {
268+ const api = repoApi(x.base, x.owner, x.repo);
269+ return { url: `${api}/milestones/${m.number}`, html_url: `${repoHtml(x.base, x.owner, x.repo)}/milestone/${m.number}`, id: m.number, number: m.number, title: m.title, description: null, state: m.state, due_on: m.dueOn };
270+}
246271 const association = (x: IssueCtx, login: string) => (login.toLowerCase() === x.owner.toLowerCase() ? "OWNER" : "NONE");
247272
248273 export function ghIssue(x: IssueCtx, i: IssueLike) {
@@ -268,7 +293,7 @@
268293 locked: !!i.locked,
269294 assignee: assignees[0] ?? null,
270295 assignees,
271- milestone: null,
296+ milestone: i.milestone ? ghIssueMilestone(x, i.milestone) : null,
272297 comments: i.comments,
273298 created_at: isoSeconds(i.createdAt),
274299 updated_at: isoSeconds(i.updatedAt),
@@ -319,7 +344,7 @@
319344 user: issue.user,
320345 body: issue.body,
321346 labels: issue.labels,
322- milestone: null,
347+ milestone: i.milestone ? ghIssueMilestone(x, i.milestone) : null,
323348 active_lock_reason: issue.active_lock_reason,
324349 created_at: issue.created_at,
325350 updated_at: issue.updated_at,
modifiedapps/diffinery/src/http/git.ts+4−0View file
@@ -2,6 +2,7 @@
22 import type { Env } from "../env.js";
33 import type { AppVars } from "../auth/principal.js";
44 import { HttpError } from "../lib/util.js";
5+import { importPushAllowed } from "../imports/service.js";
56 import { loadRepo } from "./common.js";
67
78 /**
@@ -35,6 +36,9 @@
3536
3637 git.post("/:owner/:repo/git-receive-pack", async (c) => {
3738 const { stub, caller, repo } = await loadRepo(c, c.req.param("owner"), c.req.param("repo"), "write");
39+ // A GitHub import's own token, while that import is running and filling this repository, may push in import mode.
40+ const p = c.get("principal");
41+ if (p.type === "account" && p.importId && p.tokenId && (await importPushAllowed(c.env, p.importId, repo.id, p.tokenId))) caller.importId = p.importId;
3842 const body = await readBody(c.req.raw);
3943 const out = await stub.gitReceivePack(body, caller);
4044 c.executionCtx.waitUntil(c.get("dir").touchPushed(repo.id));
addedapps/diffinery/src/http/imports.ts+367−0View file
@@ -1,0 +1,367 @@
1+import { Hono } from "hono";
2+import type { ImportGitResult, ImportProgress } from "@diffinery/shared";
3+import type { Env } from "../env.js";
4+import { credentialFrom, requireAccount, requireAccountScope, type AppContext, type AppVars } from "../auth/principal.js";
5+import type { Account } from "../lib/directory.js";
6+import { HttpError } from "../lib/util.js";
7+import { BULK_STEPS, ImportService, REPO_STEPS, type GitState, type ImportRow, type ImportSummary, type StepState, type StoredRequest } from "../imports/service.js";
8+import { webHostOf } from "../imports/options.js";
9+import type { Milestone } from "../repo/imports.js";
10+import { loadRepo } from "./common.js";
11+
12+/**
13+ * GitHub imports (`/api/v3/diffinery/imports`), the import agent's endpoints,
14+ * milestones (`/repos/:owner/:repo/milestones`, GitHub-compatible), organization
15+ * Actions variables and linked GitHub accounts. Mounted from api.ts.
16+ */
17+export const imports = new Hono<{ Bindings: Env; Variables: AppVars }>();
18+
19+async function body<T>(c: AppContext): Promise<T> {
20+ try {
21+ return (await c.req.json()) as T;
22+ } catch {
23+ throw new HttpError(400, "request body must be JSON");
24+ }
25+}
26+
27+/** A person or agent signed in normally (an import's own push token can't manage imports). */
28+function actor(c: AppContext): Account {
29+ const a = requireAccount(c);
30+ const p = c.get("principal");
31+ if (p.type === "account" && p.importId) throw new HttpError(403, "this token can only be used by its GitHub import");
32+ return a;
33+}
34+
35+const steps = (row: ImportRow) => {
36+ const state = JSON.parse(row.steps || "{}") as Record<string, StepState>;
37+ return (row.kind === "bulk" ? BULK_STEPS : REPO_STEPS).map(([name, title]) => ({ name, title, status: state[name]?.status ?? "pending", message: state[name]?.message ?? null, count: state[name]?.count ?? null, started_at: state[name]?.startedAt ?? null, finished_at: state[name]?.finishedAt ?? null }));
38+};
39+
40+/** The JSON form of an import (no credentials, ever). Shared by the REST API, MCP and the web UI. */
41+export async function importJson(env: Env, svc: ImportService, row: ImportRow, o: { children?: boolean } = {}) {
42+ const [owner, creator] = await Promise.all([svc.dir.account(row.target_owner_id), svc.dir.account(row.created_by)]);
43+ const req = JSON.parse(row.request || "{}") as StoredRequest;
44+ const summary = JSON.parse(row.summary || "{}") as ImportSummary;
45+ const git = row.git ? (JSON.parse(row.git) as GitState) : null;
46+ const web = webHostOf(row.api_url);
47+ const fullName = row.target_name ? `${owner?.login}/${row.target_name}` : null;
48+ const out = {
49+ id: row.id,
50+ url: `${env.PUBLIC_URL}/api/v3/diffinery/imports/${row.id}`,
51+ html_url: `${env.PUBLIC_URL}/imports/${row.id}`,
52+ kind: row.kind === "bulk" ? "organization" : "repository",
53+ parent_id: row.parent_id,
54+ status: row.status,
55+ attempt: row.attempt,
56+ source: { full_name: row.source, html_url: `https://${web}/${row.source}`, api_url: row.api_url },
57+ target: { owner: owner?.login ?? null, name: row.target_name, full_name: fullName, html_url: fullName && row.repo_id ? `${env.PUBLIC_URL}/${fullName}` : null },
58+ visibility: req.visibility,
59+ options: req.options ? { issues: req.options.issues, releases: req.options.releases, labels: req.options.labels, branch_protection: req.options.protection, variables: req.options.variables, lfs: req.options.lfs, secrets: req.options.secrets } : null,
60+ concurrency: row.kind === "bulk" ? req.concurrency : undefined,
61+ candidates: row.status === "draft" ? (req.candidates ?? []) : undefined,
62+ created_by: creator ? { login: creator.login, type: creator.kind === "agent" ? "Bot" : "User" } : null,
63+ steps: steps(row),
64+ git: git ? { phase: git.phase ?? null, message: git.message ?? null, chunks: git.chunks ?? 0, bytes: git.bytes ?? 0, tries: git.tries ?? 0 } : null,
65+ summary,
66+ warnings: JSON.parse(row.warnings || "[]") as string[],
67+ error: row.error,
68+ created_at: row.created_at,
69+ updated_at: row.updated_at,
70+ started_at: row.started_at,
71+ finished_at: row.finished_at,
72+ children: undefined as unknown[] | undefined,
73+ };
74+ if (o.children && row.kind === "bulk") {
75+ const owners = new Map<string, string>([[row.target_owner_id, owner?.login ?? ""]]);
76+ out.children = (await svc.children(row.id)).map((ch) => {
77+ const s = JSON.parse(ch.steps || "{}") as Record<string, StepState>;
78+ const current = REPO_STEPS.find(([n]) => s[n]?.status === "running");
79+ return {
80+ id: ch.id,
81+ html_url: `${env.PUBLIC_URL}/imports/${ch.id}`,
82+ source: ch.source,
83+ target: `${owners.get(ch.target_owner_id)}/${ch.target_name}`,
84+ status: ch.status,
85+ step: current ? current[1] : null,
86+ error: ch.error,
87+ };
88+ });
89+ }
90+ return out;
91+}
92+
93+const loadImport = async (c: AppContext, svc: ImportService, a: Account) => {
94+ const row = await svc.row(c.req.param("id")!);
95+ if (!row || !(await svc.visibleTo(row, a))) throw new HttpError(404, "import not found");
96+ return row;
97+};
98+
99+const scopesOf = (c: AppContext) => {
100+ const p = c.get("principal");
101+ return p.type === "account" ? p.scopes : [];
102+};
103+
104+// ---------------------------------------------------------------- imports
105+
106+imports.post("/diffinery/imports", async (c) => {
107+ const a = actor(c);
108+ const svc = new ImportService(c.env);
109+ const row = await svc.create(a, scopesOf(c), await body(c));
110+ return c.json(await importJson(c.env, svc, row, { children: true }), 201);
111+});
112+
113+imports.get("/diffinery/imports", async (c) => {
114+ const a = actor(c);
115+ const svc = new ImportService(c.env);
116+ const rows = await svc.list(a.id);
117+ return c.json(await Promise.all(rows.map((r) => importJson(c.env, svc, r))));
118+});
119+
120+imports.get("/diffinery/imports/:id", async (c) => {
121+ const a = actor(c);
122+ const svc = new ImportService(c.env);
123+ return c.json(await importJson(c.env, svc, await loadImport(c, svc, a), { children: true }));
124+});
125+
126+imports.post("/diffinery/imports/:id/cancel", async (c) => {
127+ requireAccountScope(c, "write");
128+ const a = actor(c);
129+ const svc = new ImportService(c.env);
130+ const row = await svc.cancel((await loadImport(c, svc, a)).id, a);
131+ return c.json(await importJson(c.env, svc, row, { children: true }));
132+});
133+
134+imports.post("/diffinery/imports/:id/retry", async (c) => {
135+ requireAccountScope(c, "write");
136+ const a = actor(c);
137+ const svc = new ImportService(c.env);
138+ const b = await body<{ github_token?: string | null }>(c).catch(() => ({}) as { github_token?: string | null });
139+ const row = await svc.retry((await loadImport(c, svc, a)).id, a, typeof b.github_token === "string" && b.github_token.trim() ? b.github_token.trim() : null);
140+ return c.json(await importJson(c.env, svc, row, { children: true }), 202);
141+});
142+
143+/** Bulk drafts: choose the repositories, which starts the import. */
144+imports.post("/diffinery/imports/:id/selection", async (c) => {
145+ requireAccountScope(c, "write");
146+ const a = actor(c);
147+ const svc = new ImportService(c.env);
148+ const b = await body<{ repositories?: unknown }>(c);
149+ if (!Array.isArray(b.repositories) || b.repositories.some((r) => typeof r !== "string")) throw new HttpError(422, "repositories must be a list of names");
150+ const row = await svc.select(c.req.param("id"), a, b.repositories as string[]);
151+ return c.json(await importJson(c.env, svc, row, { children: true }), 202);
152+});
153+
154+/** Local development and tests (IMPORT_GIT_AGENT=external): hand the Git transfer to an agent started by hand. */
155+imports.post("/diffinery/imports/:id/claim", async (c) => {
156+ const a = actor(c);
157+ return c.json(await new ImportService(c.env).claim(c.req.param("id"), a));
158+});
159+
160+// ---------------------------------------------------------------- the import agent (push token only)
161+
162+const agentAuth = (c: AppContext): { importId: string | undefined; tokenId: string | undefined } => {
163+ const p = c.get("principal");
164+ return p.type === "account" ? { importId: p.importId, tokenId: p.tokenId } : { importId: undefined, tokenId: undefined };
165+};
166+
167+imports.get("/diffinery/imports/:id/agent/spec", async (c) => {
168+ const a = agentAuth(c);
169+ return c.json(await new ImportService(c.env).agentSpec(c.req.param("id"), a.importId, a.tokenId, credentialFrom(c.req.raw) ?? ""));
170+});
171+
172+imports.post("/diffinery/imports/:id/agent/progress", async (c) => {
173+ const a = agentAuth(c);
174+ await new ImportService(c.env).agentProgress(c.req.param("id"), a.importId, a.tokenId, await body<ImportProgress>(c));
175+ return c.body(null, 204);
176+});
177+
178+imports.post("/diffinery/imports/:id/agent/complete", async (c) => {
179+ const a = agentAuth(c);
180+ await new ImportService(c.env).agentComplete(c.req.param("id"), a.importId, a.tokenId, await body<ImportGitResult>(c));
181+ return c.body(null, 204);
182+});
183+
184+// ---------------------------------------------------------------- linked GitHub accounts
185+
186+imports.get("/diffinery/user/linked-accounts", async (c) => c.json(await new ImportService(c.env).links(actor(c).id)));
187+
188+/** Step 1 of gist linking: the nonce the person must put in a PUBLIC gist. */
189+imports.post("/diffinery/user/linked-accounts/github/challenge", async (c) => {
190+ requireAccountScope(c, "admin");
191+ const a = actor(c);
192+ const b = await body<{ api_url?: string }>(c).catch(() => ({}) as { api_url?: string });
193+ return c.json(await new ImportService(c.env).linkChallenge(a, b.api_url ?? null));
194+});
195+
196+/** Step 2: verify the gist contains the nonce and link the account. */
197+imports.post("/diffinery/user/linked-accounts/github", async (c) => {
198+ requireAccountScope(c, "admin");
199+ const a = actor(c);
200+ const b = await body<{ login?: string; api_url?: string }>(c);
201+ if (!b.login || typeof b.login !== "string") throw new HttpError(422, "login (your GitHub username) is required");
202+ return c.json(await new ImportService(c.env).linkByGist(a, b.login.trim(), b.api_url ?? null), 201);
203+});
204+
205+imports.delete("/diffinery/user/linked-accounts/:id", async (c) => {
206+ requireAccountScope(c, "admin");
207+ await new ImportService(c.env).unlink(actor(c), c.req.param("id"));
208+ return c.body(null, 204);
209+});
210+
211+/** Site admin: clear any account's GitHub link. */
212+imports.delete("/diffinery/admin/linked-accounts/:id", async (c) => {
213+ requireAccountScope(c, "admin");
214+ const a = actor(c);
215+ if (!a.site_admin) throw new HttpError(403, "site admin only");
216+ await new ImportService(c.env).clearLink(a, c.req.param("id"));
217+ return c.body(null, 204);
218+});
219+
220+// ---------------------------------------------------------------- milestones
221+
222+const P = (c: AppContext) => ({ owner: c.req.param("owner")!, repo: c.req.param("repo")! });
223+const num = (c: AppContext, name = "number") => {
224+ const n = Number(c.req.param(name));
225+ if (!Number.isInteger(n) || n < 1) throw new HttpError(404, "not found");
226+ return n;
227+};
228+
229+
230+const milestoneJson = (c: AppContext, m: Milestone) => ({
231+ url: `${c.env.PUBLIC_URL}/api/v3/repos/${P(c).owner}/${P(c).repo}/milestones/${m.number}`,
232+ html_url: `${c.env.PUBLIC_URL}/${P(c).owner}/${P(c).repo}/issues?milestone=${m.number}`,
233+ id: m.number,
234+ number: m.number,
235+ title: m.title,
236+ description: m.description,
237+ state: m.state,
238+ creator: m.creator ? { login: m.creator } : null,
239+ open_issues: m.openIssues,
240+ closed_issues: m.closedIssues,
241+ created_at: m.createdAt,
242+ updated_at: m.updatedAt,
243+ closed_at: m.closedAt,
244+ due_on: m.dueOn,
245+});
246+
247+const milestoneInput = (b: Record<string, unknown>) => {
248+ const out: { title?: string; description?: string; state?: "open" | "closed"; dueOn?: string | null } = {};
249+ if (b.title !== undefined) {
250+ if (typeof b.title !== "string" || !b.title.trim() || b.title.length > 255) throw new HttpError(422, "title must be 1 to 255 characters");
251+ out.title = b.title;
252+ }
253+ if (b.description !== undefined && b.description !== null) {
254+ if (typeof b.description !== "string" || b.description.length > 65536) throw new HttpError(422, "description must be a string");
255+ out.description = b.description;
256+ }
257+ if (b.state !== undefined) {
258+ if (b.state !== "open" && b.state !== "closed") throw new HttpError(422, "state must be open or closed");
259+ out.state = b.state;
260+ }
261+ if (b.due_on !== undefined) {
262+ if (b.due_on !== null && (typeof b.due_on !== "string" || Number.isNaN(Date.parse(b.due_on)))) throw new HttpError(422, "due_on must be an ISO 8601 timestamp");
263+ out.dueOn = b.due_on === null ? null : new Date(b.due_on as string).toISOString();
264+ }
265+ return out;
266+};
267+
268+imports.get("/repos/:owner/:repo/milestones", async (c) => {
269+ const { stub } = await loadRepo(c, P(c).owner, P(c).repo);
270+ const state = c.req.query("state") ?? "open";
271+ if (!["open", "closed", "all"].includes(state)) throw new HttpError(422, "state must be open, closed or all");
272+ const list = await stub.milestones(state as "open" | "closed" | "all");
273+ return c.json(list.map((m) => milestoneJson(c, m)));
274+});
275+
276+imports.get("/repos/:owner/:repo/milestones/:number", async (c) => {
277+ const { stub } = await loadRepo(c, P(c).owner, P(c).repo);
278+ return c.json(milestoneJson(c, await stub.milestone(num(c))));
279+});
280+
281+imports.post("/repos/:owner/:repo/milestones", async (c) => {
282+ const { stub, caller } = await loadRepo(c, P(c).owner, P(c).repo, "triage");
283+ const input = milestoneInput(await body<Record<string, unknown>>(c));
284+ if (!input.title) throw new HttpError(422, "title is required");
285+ return c.json(milestoneJson(c, await stub.saveMilestone(null, input, caller.login)), 201);
286+});
287+
288+imports.patch("/repos/:owner/:repo/milestones/:number", async (c) => {
289+ const { stub, caller } = await loadRepo(c, P(c).owner, P(c).repo, "triage");
290+ return c.json(milestoneJson(c, await stub.saveMilestone(num(c), milestoneInput(await body<Record<string, unknown>>(c)), caller.login)));
291+});
292+
293+// `PATCH /issues/:number` with `milestone` (a number or null), as on GitHub. The rest of the
294+// request is handled by the issue route in api.ts, which runs next.
295+imports.patch("/repos/:owner/:repo/issues/:number", async (c, next) => {
296+ const b = await body<{ milestone?: unknown }>(c);
297+ let milestone: number | null | undefined;
298+ if (b.milestone !== undefined) {
299+ if (b.milestone !== null && (typeof b.milestone !== "number" || !Number.isInteger(b.milestone))) throw new HttpError(422, "milestone must be a milestone number or null");
300+ milestone = b.milestone as number | null;
301+ }
302+ // Let the main issue handler (api.ts) validate and apply first; only change the milestone if it succeeded,
303+ // so a rejected PATCH never leaves the milestone half-changed.
304+ await next();
305+ if (milestone !== undefined && c.res.status < 400) {
306+ const { stub } = await loadRepo(c, P(c).owner, P(c).repo, "triage");
307+ await stub.setIssueMilestone(num(c), milestone);
308+ }
309+});
310+
311+// ---------------------------------------------------------------- organization Actions variables
312+
313+async function orgFor(c: AppContext, admin: boolean) {
314+ const a = actor(c);
315+ const dir = c.get("dir");
316+ const org = await dir.accountByLogin(c.req.param("org")!);
317+ if (!org || org.kind !== "org") throw new HttpError(404, "not found");
318+ const role = await dir.orgRole(org.id, a.kind === "agent" ? (a.owner_id ?? "") : a.id);
319+ const isMember = !!role || a.site_admin || (a.kind === "agent" && a.owner_id === org.id);
320+ if (!isMember) throw new HttpError(404, "not found");
321+ if (admin && (a.kind === "agent" || (role !== "owner" && role !== "admin" && !a.site_admin))) throw new HttpError(403, "organization admins only");
322+ return { a, org };
323+}
324+
325+const VAR_NAME = /^[A-Za-z_][A-Za-z0-9_]*$/;
326+
327+imports.get("/orgs/:org/actions/variables", async (c) => {
328+ // Low 2: variable values are readable only by an org owner/admin, not every member.
329+ const { org } = await orgFor(c, true);
330+ const vars = await c.get("dir").orgVariables(org.id);
331+ return c.json({ total_count: vars.length, variables: vars.map((v) => ({ name: v.name, value: v.value, visibility: "all", updated_at: v.updated_at, created_at: v.updated_at })) });
332+});
333+
334+const putOrgVariable = async (c: AppContext, name: string, value: unknown) => {
335+ requireAccountScope(c, "admin");
336+ const { a, org } = await orgFor(c, true);
337+ if (!VAR_NAME.test(name) || /^GITHUB_/i.test(name)) throw new HttpError(422, "invalid variable name");
338+ if (typeof value !== "string" || value.length > 48 * 1024) throw new HttpError(422, "value must be a string of at most 48 KB");
339+ await c.get("dir").setOrgVariables(org.id, { [name]: value });
340+ c.executionCtx.waitUntil(c.get("dir").audit({ actorId: a.id, actorLogin: a.login, action: "org.variable_set", target: org.login, detail: { name: name.toUpperCase() } }));
341+};
342+
343+imports.post("/orgs/:org/actions/variables", async (c) => {
344+ const b = await body<{ name?: string; value?: string }>(c);
345+ await putOrgVariable(c, String(b.name ?? ""), b.value);
346+ return c.body(null, 201);
347+});
348+
349+imports.patch("/orgs/:org/actions/variables/:name", async (c) => {
350+ await putOrgVariable(c, c.req.param("name"), (await body<{ value?: string }>(c)).value);
351+ return c.body(null, 204);
352+});
353+
354+imports.get("/orgs/:org/actions/variables/:name", async (c) => {
355+ const { org } = await orgFor(c, true);
356+ const v = (await c.get("dir").orgVariables(org.id)).find((x) => x.name === c.req.param("name").toUpperCase());
357+ if (!v) throw new HttpError(404, "variable not found");
358+ return c.json({ name: v.name, value: v.value, visibility: "all", updated_at: v.updated_at, created_at: v.updated_at });
359+});
360+
361+imports.delete("/orgs/:org/actions/variables/:name", async (c) => {
362+ requireAccountScope(c, "admin");
363+ const { a, org } = await orgFor(c, true);
364+ await c.get("dir").setOrgVariables(org.id, { [c.req.param("name")]: null });
365+ c.executionCtx.waitUntil(c.get("dir").audit({ actorId: a.id, actorLogin: a.login, action: "org.variable_delete", target: org.login, detail: { name: c.req.param("name").toUpperCase() } }));
366+ return c.body(null, 204);
367+});
addedapps/diffinery/src/imports/container.ts+48−0View file
@@ -1,0 +1,48 @@
1+import { Container } from "@cloudflare/containers";
2+import type { StopParams } from "@cloudflare/containers";
3+import type { Env } from "../env.js";
4+import { ImportService } from "./service.js";
5+
6+/**
7+ * One container per GitHub import's Git transfer: the runner image in import
8+ * mode (`DIFFINERY_IMPORT_ID`), which clones from GitHub, pushes to Diffinery in
9+ * size-bounded chunks and reports back through the import's agent endpoints.
10+ */
11+export class ImportContainer extends Container<Env> {
12+ // The agent makes no inbound requests, so activity-based sleep is the only timer; a little over the 6h transfer cap.
13+ override sleepAfter = "7h";
14+ override enableInternet = true;
15+
16+ async startImport(input: { importId: string; token: string; apiUrl: string; chunkBytes?: number; stamp?: string }): Promise<void> {
17+ await this.ctx.storage.put("import", input.importId);
18+ // The attempt-try this container serves, so a late stop of an old try is ignored (M4 / Correctness 7).
19+ if (input.stamp) await this.ctx.storage.put("stamp", input.stamp);
20+ await this.start({
21+ envVars: {
22+ DIFFINERY_API_URL: input.apiUrl,
23+ DIFFINERY_IMPORT_ID: input.importId,
24+ DIFFINERY_IMPORT_TOKEN: input.token,
25+ ...(input.chunkBytes ? { DIFFINERY_IMPORT_CHUNK_BYTES: String(input.chunkBytes) } : {}),
26+ },
27+ });
28+ }
29+
30+ async stopImport(): Promise<void> {
31+ if (this.ctx.container?.running) await this.stop("SIGTERM");
32+ }
33+
34+ override async onStop(params: StopParams): Promise<void> {
35+ const importId = await this.ctx.storage.get<string>("import");
36+ if (!importId) return;
37+ const stamp = await this.ctx.storage.get<string>("stamp");
38+ // A transfer that stopped without reporting (crash, out of memory, platform restart) fails so the workflow can retry it;
39+ // a stale stamp (an old try stopping late) is ignored by the service.
40+ await new ImportService(this.env).gitStopped(importId, params.exitCode ?? null, stamp);
41+ await this.ctx.storage.delete("import");
42+ await this.ctx.storage.delete("stamp");
43+ }
44+
45+ override onError(error: unknown): void {
46+ console.error("import container error", error);
47+ }
48+}
addedapps/diffinery/src/imports/github.ts+173−0View file
@@ -1,0 +1,173 @@
1+/**
2+ * A small GitHub REST client for imports: GET only, `Link` pagination, rate
3+ * limits surfaced as `RateLimited` so the workflow can sleep instead of
4+ * failing, and redirects followed only on the same host (no SSRF through a
5+ * redirect).
6+ */
7+
8+export class RateLimited extends Error {
9+ constructor(
10+ /** Seconds to wait before trying again. */
11+ readonly waitSeconds: number,
12+ ) {
13+ super(`GitHub rate limit reached; continuing in ${waitSeconds} seconds`);
14+ }
15+}
16+
17+export class GitHubError extends Error {
18+ constructor(
19+ readonly status: number,
20+ message: string,
21+ ) {
22+ super(message);
23+ }
24+}
25+
26+export interface Page<T> {
27+ items: T[];
28+ /** The next page as a path relative to the API base, or null on the last page. */
29+ next: string | null;
30+}
31+
32+/** Join byte chunks into one array of a known total length. */
33+function concat(chunks: Uint8Array[], total: number): Uint8Array {
34+ const out = new Uint8Array(total);
35+ let at = 0;
36+ for (const c of chunks) {
37+ out.set(c, at);
38+ at += c.byteLength;
39+ }
40+ return out;
41+}
42+
43+export class GitHubClient {
44+ /** Never buffer more than this from a GitHub JSON response (API payloads are far smaller). */
45+ static readonly MAX_JSON_BYTES = 8 * 1024 * 1024;
46+
47+ private readonly base: URL;
48+
49+ constructor(
50+ apiUrl: string,
51+ private readonly token: string | null,
52+ private readonly fetcher: typeof fetch = (input, init) => fetch(input, init),
53+ ) {
54+ this.base = new URL(apiUrl.replace(/\/+$/, "") + "/");
55+ }
56+
57+ /** `path` is relative to the API base ("repos/o/r/issues?state=all") or an absolute URL on the same host. */
58+ private resolve(path: string): URL {
59+ const u = new URL(path.replace(/^\/+/, ""), this.base);
60+ if (u.origin !== this.base.origin || !u.pathname.startsWith(this.base.pathname)) throw new GitHubError(400, "GitHub returned a link to another host");
61+ return u;
62+ }
63+
64+ /** The part of a same-host URL after the API base, for storing as a cursor. */
65+ private relative(u: URL): string {
66+ return u.pathname.slice(this.base.pathname.length) + u.search;
67+ }
68+
69+ private async request(path: string, init: { method?: string; body?: unknown } = {}): Promise<Response> {
70+ let url = this.resolve(path);
71+ for (let hop = 0; hop < 4; hop++) {
72+ const headers: Record<string, string> = {
73+ accept: "application/vnd.github+json",
74+ "user-agent": "Diffinery-Importer",
75+ "x-github-api-version": "2022-11-28",
76+ };
77+ if (this.token) headers.authorization = `Bearer ${this.token}`;
78+ const method = init.method ?? "GET";
79+ let body: string | undefined;
80+ if (init.body !== undefined) {
81+ body = JSON.stringify(init.body);
82+ headers["content-type"] = "application/json";
83+ }
84+ const res = await this.fetcher(url.toString(), { method, body, headers, redirect: "manual", signal: AbortSignal.timeout(30_000) });
85+ if ([301, 302, 307, 308].includes(res.status) && method === "GET") {
86+ const loc = res.headers.get("location");
87+ if (!loc) throw new GitHubError(502, "GitHub redirected without a location");
88+ url = this.resolve(new URL(loc, url).toString());
89+ continue;
90+ }
91+ // Primary limits answer 403 with remaining 0; secondary limits answer 403 or 429 with retry-after.
92+ if (res.status === 429 || (res.status === 403 && (res.headers.get("x-ratelimit-remaining") === "0" || res.headers.has("retry-after")))) {
93+ const retry = Number(res.headers.get("retry-after"));
94+ const reset = Number(res.headers.get("x-ratelimit-reset"));
95+ const wait = retry > 0 ? retry : reset > 0 ? reset - Math.floor(Date.now() / 1000) + 5 : 60;
96+ throw new RateLimited(Math.min(Math.max(wait, 5), 3600));
97+ }
98+ if (!res.ok) {
99+ const text = await res.text().catch(() => "");
100+ let message = text.slice(0, 200);
101+ try {
102+ message = (JSON.parse(text) as { message?: string }).message ?? message;
103+ } catch {
104+ // not JSON
105+ }
106+ // Secondary (abuse) rate limits come back as 403/422 with a distinctive message; treat them
107+ // as a rate limit to wait on, not a fatal error.
108+ if (/secondary rate limit|abuse detection|you have exceeded a secondary/i.test(message)) {
109+ const retry = Number(res.headers.get("retry-after"));
110+ throw new RateLimited(Math.min(Math.max(retry > 0 ? retry : 60, 5), 3600));
111+ }
112+ throw new GitHubError(res.status, `GitHub ${res.status} for ${url.pathname}: ${message}`);
113+ }
114+ return res;
115+ }
116+ throw new GitHubError(508, "too many redirects");
117+ }
118+
119+ /**
120+ * Read and parse a JSON body, never buffering more than `MAX_JSON_BYTES`.
121+ * API responses are small, but a bounded read keeps a hostile or broken
122+ * server from exhausting memory in a Worker or Durable Object.
123+ */
124+ private async readJson<T>(res: Response): Promise<T> {
125+ const max = GitHubClient.MAX_JSON_BYTES;
126+ const declared = Number(res.headers.get("content-length"));
127+ if (declared > max) throw new GitHubError(502, `GitHub response is too large (${declared} bytes)`);
128+ const body = res.body;
129+ if (!body) return JSON.parse(await res.text()) as T;
130+ const reader = body.getReader();
131+ const chunks: Uint8Array[] = [];
132+ let total = 0;
133+ for (;;) {
134+ const { done, value } = await reader.read();
135+ if (done) break;
136+ total += value.byteLength;
137+ if (total > max) {
138+ await reader.cancel().catch(() => {});
139+ throw new GitHubError(502, "GitHub response exceeded the size limit");
140+ }
141+ chunks.push(value);
142+ }
143+ return JSON.parse(new TextDecoder().decode(concat(chunks, total))) as T;
144+ }
145+
146+ async get<T>(path: string): Promise<T> {
147+ return this.readJson<T>(await this.request(path));
148+ }
149+
150+ /** One page of a list endpoint. */
151+ async page<T>(path: string): Promise<Page<T>> {
152+ const res = await this.request(path);
153+ const items = await this.readJson<T[] | { repositories?: T[]; variables?: T[]; secrets?: T[]; environments?: T[]; branch_policies?: T[] }>(res);
154+ const next = /<([^>]+)>;\s*rel="next"/.exec(res.headers.get("link") ?? "")?.[1];
155+ const list = Array.isArray(items) ? items : (items.repositories ?? items.variables ?? items.secrets ?? items.environments ?? items.branch_policies ?? []);
156+ return { items: list, next: next ? this.relative(this.resolve(next)) : null };
157+ }
158+
159+ /** Every item of a list endpoint (for short lists: labels, milestones, variables). */
160+ async all<T>(path: string, max = 5000): Promise<T[]> {
161+ const out: T[] = [];
162+ let next: string | null = path;
163+ while (next && out.length < max) {
164+ const p: Page<T> = await this.page<T>(next);
165+ out.push(...p.items);
166+ next = p.next;
167+ }
168+ return out;
169+ }
170+}
171+
172+/** `path?query` with `per_page=100` added. */
173+export const listPath = (path: string, query: Record<string, string> = {}) => `${path}?${new URLSearchParams({ per_page: "100", ...query })}`;
addedapps/diffinery/src/imports/mapping.ts+351−0View file
@@ -1,0 +1,351 @@
1+/**
2+ * Mapping GitHub REST objects to the rows an import writes into a repository.
3+ * Pure functions, unit tested; the workflow fetches and the repository stores.
4+ */
5+
6+export interface GhUser {
7+ login: string;
8+ id?: number;
9+ type?: string;
10+}
11+
12+export interface GhLabel {
13+ name: string;
14+ color?: string | null;
15+ description?: string | null;
16+}
17+
18+export interface GhMilestone {
19+ number: number;
20+ title: string;
21+ description?: string | null;
22+ state: "open" | "closed";
23+ due_on?: string | null;
24+ created_at: string;
25+ updated_at: string;
26+ closed_at?: string | null;
27+ creator?: GhUser | null;
28+}
29+
30+export interface GhIssue {
31+ number: number;
32+ title: string;
33+ body?: string | null;
34+ state: "open" | "closed";
35+ state_reason?: string | null;
36+ user: GhUser | null;
37+ labels?: Array<GhLabel | string>;
38+ assignees?: GhUser[] | null;
39+ milestone?: { number: number } | null;
40+ locked?: boolean;
41+ created_at: string;
42+ updated_at: string;
43+ closed_at?: string | null;
44+ pull_request?: unknown;
45+}
46+
47+export interface GhPull extends GhIssue {
48+ draft?: boolean;
49+ merged_at?: string | null;
50+ merge_commit_sha?: string | null;
51+ merged_by?: GhUser | null;
52+ requested_reviewers?: GhUser[] | null;
53+ head: { ref: string; sha: string; label?: string; repo: { full_name: string } | null };
54+ base: { ref: string; sha?: string; repo?: { full_name: string } | null };
55+}
56+
57+export interface GhComment {
58+ id: number;
59+ body?: string | null;
60+ user: GhUser | null;
61+ created_at: string;
62+ updated_at: string;
63+ issue_url?: string;
64+}
65+
66+export interface GhReview {
67+ id: number;
68+ user: GhUser | null;
69+ state: string;
70+ body?: string | null;
71+ commit_id?: string | null;
72+ submitted_at?: string | null;
73+}
74+
75+export interface GhReviewComment {
76+ id: number;
77+ pull_request_review_id?: number | null;
78+ pull_request_url?: string;
79+ user: GhUser | null;
80+ body?: string | null;
81+ path: string;
82+ line?: number | null;
83+ original_line?: number | null;
84+ side?: "LEFT" | "RIGHT" | null;
85+ commit_id?: string | null;
86+ original_commit_id?: string | null;
87+ in_reply_to_id?: number | null;
88+ created_at: string;
89+ updated_at: string;
90+}
91+
92+/** Who did something: a linked Diffinery account, or a GitHub user shown as imported. */
93+export interface Person {
94+ id: string;
95+ login: string;
96+ kind: "user" | "github";
97+}
98+
99+/** A verified link from a GitHub account to a Diffinery account (see account_links). */
100+export interface Link {
101+ externalId: string;
102+ externalLogin: string;
103+ accountId: string;
104+ login: string;
105+}
106+
107+/** Prefix for GitHub logins stored where Diffinery expects a login (assignees, reviewers, merged by). */
108+export const IMPORTED_PREFIX = "github:";
109+
110+/**
111+ * Maps GitHub users to Diffinery accounts only through verified links, matched by
112+ * GitHub's numeric user id (logins can be renamed and reused). Everyone else stays
113+ * a GitHub user, shown as imported; no accounts are created.
114+ */
115+export class People {
116+ private byId = new Map<string, Link>();
117+ private byLogin = new Map<string, Link>();
118+ readonly mapped = new Set<string>();
119+ readonly unmapped = new Set<string>();
120+
121+ constructor(links: Link[]) {
122+ for (const l of links) {
123+ this.byId.set(l.externalId, l);
124+ this.byLogin.set(l.externalLogin.toLowerCase(), l);
125+ }
126+ }
127+
128+ private link(u: GhUser): Link | undefined {
129+ // Objects without an id (rare, e.g. some Enterprise payloads) fall back to the linked login.
130+ return u.id !== undefined && u.id !== null ? this.byId.get(String(u.id)) : this.byLogin.get(u.login.toLowerCase());
131+ }
132+
133+ person(u: GhUser | null | undefined): Person {
134+ if (!u) return { id: "github:ghost", login: "ghost", kind: "github" };
135+ const l = this.link(u);
136+ if (l) {
137+ this.mapped.add(u.login);
138+ return { id: l.accountId, login: l.login, kind: "user" };
139+ }
140+ this.unmapped.add(u.login);
141+ return { id: `github:${u.id ?? u.login}`, login: u.login, kind: "github" };
142+ }
143+
144+ /** A login for fields that hold logins: the Diffinery login when linked, else "github:<login>". */
145+ login(u: GhUser | null | undefined): string {
146+ const p = this.person(u);
147+ return p.kind === "user" ? p.login : `${IMPORTED_PREFIX}${p.login}`;
148+ }
149+}
150+
151+export interface ImportedIssue {
152+ number: number;
153+ kind: "issue" | "pull";
154+ title: string;
155+ body: string;
156+ state: "open" | "closed";
157+ stateReason: string | null;
158+ author: Person;
159+ assignees: string[];
160+ labels: string[];
161+ milestone: number | null;
162+ locked: boolean;
163+ createdAt: string;
164+ updatedAt: string;
165+ closedAt: string | null;
166+ pull?: {
167+ headRef: string;
168+ headSha: string;
169+ baseRef: string;
170+ draft: boolean;
171+ merged: boolean;
172+ mergedAt: string | null;
173+ mergedBy: string | null;
174+ mergeCommit: string | null;
175+ requestedReviewers: string[];
176+ };
177+}
178+
179+export interface ImportedComment {
180+ id: string;
181+ number: number;
182+ author: Person;
183+ body: string;
184+ createdAt: string;
185+ updatedAt: string;
186+ reviewId?: string | null;
187+ path?: string | null;
188+ line?: number | null;
189+ side?: "LEFT" | "RIGHT" | null;
190+ commitId?: string | null;
191+ inReplyTo?: string | null;
192+}
193+
194+export interface ImportedReview {
195+ id: string;
196+ number: number;
197+ author: Person;
198+ state: "APPROVED" | "CHANGES_REQUESTED" | "COMMENTED" | "DISMISSED";
199+ body: string;
200+ commitId: string;
201+ submittedAt: string;
202+}
203+
204+export interface ImportedMilestone {
205+ number: number;
206+ title: string;
207+ description: string;
208+ state: "open" | "closed";
209+ dueOn: string | null;
210+ createdAt: string;
211+ updatedAt: string;
212+ closedAt: string | null;
213+ creator: string | null;
214+}
215+
216+const labelName = (l: GhLabel | string) => (typeof l === "string" ? l : l.name);
217+
218+const STATE_REASONS = new Set(["completed", "not_planned", "reopened", "duplicate"]);
219+
220+export function mapIssue(i: GhIssue, people: People): ImportedIssue {
221+ return {
222+ number: i.number,
223+ kind: "issue",
224+ title: i.title || `Issue #${i.number}`,
225+ body: i.body ?? "",
226+ state: i.state === "closed" ? "closed" : "open",
227+ stateReason: i.state === "closed" ? (i.state_reason && STATE_REASONS.has(i.state_reason) ? i.state_reason : "completed") : null,
228+ author: people.person(i.user),
229+ assignees: (i.assignees ?? []).map((a) => people.login(a)),
230+ labels: (i.labels ?? []).map(labelName).filter(Boolean),
231+ milestone: i.milestone?.number ?? null,
232+ locked: !!i.locked,
233+ createdAt: i.created_at,
234+ updatedAt: i.updated_at,
235+ closedAt: i.state === "closed" ? (i.closed_at ?? i.updated_at) : null,
236+ };
237+}
238+
239+/**
240+ * A pull request keeps its number, state and history. Same-repository heads keep
241+ * their branch name; heads in forks have no branch here, so they are kept as
242+ * "fork-owner:branch" (never a valid local branch, so pushes never move them)
243+ * with the head commit pinned by `refs/pull/N/head`.
244+ */
245+export function mapPull(p: GhPull, sourceFullName: string, people: People): ImportedIssue {
246+ const merged = !!p.merged_at;
247+ const sameRepo = p.head.repo?.full_name?.toLowerCase() === sourceFullName.toLowerCase();
248+ const forkOwner = p.head.label?.split(":")[0] ?? p.head.repo?.full_name?.split("/")[0] ?? "unknown";
249+ return {
250+ ...mapIssue(p, people),
251+ kind: "pull",
252+ title: p.title || `Pull request #${p.number}`,
253+ stateReason: p.state === "closed" ? (merged ? "merged" : null) : null,
254+ pull: {
255+ headRef: sameRepo ? p.head.ref : `${forkOwner}:${p.head.ref}`,
256+ headSha: p.head.sha,
257+ baseRef: p.base.ref,
258+ draft: !!p.draft,
259+ merged,
260+ mergedAt: p.merged_at ?? null,
261+ mergedBy: merged && p.merged_by ? people.login(p.merged_by) : null,
262+ mergeCommit: merged ? (p.merge_commit_sha ?? null) : null,
263+ requestedReviewers: p.state === "open" ? (p.requested_reviewers ?? []).map((r) => people.login(r)) : [],
264+ },
265+ };
266+}
267+
268+/** The issue or pull request number a comment belongs to, from its API URL. */
269+export function numberFromUrl(url: string | undefined): number | null {
270+ const m = /\/(?:issues|pulls)\/(\d+)$/.exec(url ?? "");
271+ return m ? Number(m[1]) : null;
272+}
273+
274+export function mapIssueComment(c: GhComment, people: People): ImportedComment | null {
275+ const number = numberFromUrl(c.issue_url);
276+ if (!number) return null;
277+ return { id: `gh-ic${c.id}`, number, author: people.person(c.user), body: c.body ?? "", createdAt: c.created_at, updatedAt: c.updated_at };
278+}
279+
280+export function mapReview(r: GhReview, number: number, people: People): ImportedReview | null {
281+ if (!r.submitted_at || r.state === "PENDING") return null;
282+ const state = (["APPROVED", "CHANGES_REQUESTED", "COMMENTED", "DISMISSED"] as const).find((s) => s === r.state) ?? "COMMENTED";
283+ return { id: `gh-r${r.id}`, number, author: people.person(r.user), state, body: r.body ?? "", commitId: r.commit_id ?? "", submittedAt: r.submitted_at };
284+}
285+
286+export function mapReviewComment(c: GhReviewComment, people: People): ImportedComment | null {
287+ const number = numberFromUrl(c.pull_request_url);
288+ if (!number) return null;
289+ return {
290+ id: `gh-rc${c.id}`,
291+ number,
292+ author: people.person(c.user),
293+ body: c.body ?? "",
294+ createdAt: c.created_at,
295+ updatedAt: c.updated_at,
296+ reviewId: c.pull_request_review_id ? `gh-r${c.pull_request_review_id}` : null,
297+ path: c.path,
298+ line: c.line ?? c.original_line ?? null,
299+ side: c.side === "LEFT" ? "LEFT" : "RIGHT",
300+ commitId: c.commit_id ?? c.original_commit_id ?? null,
301+ inReplyTo: c.in_reply_to_id ? `gh-rc${c.in_reply_to_id}` : null,
302+ };
303+}
304+
305+export function mapMilestone(m: GhMilestone, people: People): ImportedMilestone {
306+ return {
307+ number: m.number,
308+ title: m.title,
309+ description: m.description ?? "",
310+ state: m.state === "closed" ? "closed" : "open",
311+ dueOn: m.due_on ?? null,
312+ createdAt: m.created_at,
313+ updatedAt: m.updated_at,
314+ closedAt: m.closed_at ?? null,
315+ creator: m.creator ? people.login(m.creator) : null,
316+ };
317+}
318+
319+/** GitHub branch protection (REST shape) to Diffinery's rules. */
320+export interface GhProtection {
321+ required_status_checks?: { contexts?: string[]; checks?: Array<{ context: string }> } | null;
322+ required_pull_request_reviews?: { required_approving_review_count?: number; dismiss_stale_reviews?: boolean } | null;
323+ enforce_admins?: { enabled?: boolean } | null;
324+ required_linear_history?: { enabled?: boolean } | null;
325+ allow_force_pushes?: { enabled?: boolean } | null;
326+ allow_deletions?: { enabled?: boolean } | null;
327+}
328+
329+export function mapProtection(p: GhProtection): {
330+ requirePullRequest: boolean;
331+ requiredApprovals: number;
332+ dismissStaleReviews: boolean;
333+ requiredChecks: string[];
334+ allowForcePush: boolean;
335+ allowDeletion: boolean;
336+ requireLinearHistory: boolean;
337+ enforceAdmins: boolean;
338+} {
339+ const reviews = p.required_pull_request_reviews ?? null;
340+ const checks = [...(p.required_status_checks?.contexts ?? []), ...(p.required_status_checks?.checks ?? []).map((c) => c.context)];
341+ return {
342+ requirePullRequest: !!reviews,
343+ requiredApprovals: reviews?.required_approving_review_count ?? 0,
344+ dismissStaleReviews: !!reviews?.dismiss_stale_reviews,
345+ requiredChecks: [...new Set(checks)],
346+ allowForcePush: !!p.allow_force_pushes?.enabled,
347+ allowDeletion: !!p.allow_deletions?.enabled,
348+ requireLinearHistory: !!p.required_linear_history?.enabled,
349+ enforceAdmins: !!p.enforce_admins?.enabled,
350+ };
351+}
addedapps/diffinery/src/imports/options.ts+278−0View file
@@ -1,0 +1,278 @@
1+import { HttpError, isValidName } from "../lib/util.js";
2+
3+/**
4+ * Input of a GitHub import, validated and normalised. Kept free of platform
5+ * imports so it can be unit tested.
6+ */
7+
8+export type ImportVisibility = "keep" | "private" | "public";
9+
10+export interface ImportOptions {
11+ /** Issues and pull requests (with comments and reviews). */
12+ issues: boolean;
13+ releases: boolean;
14+ /** Labels and milestones. */
15+ labels: boolean;
16+ /** Branch protection rules. */
17+ protection: boolean;
18+ /** Actions variables and secret names. */
19+ variables: boolean;
20+ /** Git LFS objects. */
21+ lfs: boolean;
22+ /**
23+ * Push protection findings in old history: "bypass" imports them with an audited
24+ * admin bypass (the history is already on GitHub); "stop" fails the import.
25+ */
26+ secrets: "bypass" | "stop";
27+}
28+
29+export const DEFAULT_OPTIONS: ImportOptions = { issues: true, releases: true, labels: true, protection: true, variables: true, lfs: true, secrets: "bypass" };
30+
31+export interface ImportRequest {
32+ mode: "repo" | "bulk";
33+ /** "owner/name" for one repository, the GitHub organization or user for a bulk import. */
34+ source: string;
35+ /** Bulk: repository names to import (null: all of them). */
36+ repositories: string[] | null;
37+ excludeForks: boolean;
38+ excludeArchived: boolean;
39+ /** Diffinery user or organization that will own the repositories. */
40+ owner: string;
41+ /** Repository imports: the Diffinery repository name (default: the GitHub name). */
42+ name: string | null;
43+ visibility: ImportVisibility;
44+ options: ImportOptions;
45+ /** GitHub REST API base, e.g. https://api.github.com or https://ghe.example.com/api/v3 */
46+ apiUrl: string;
47+ token: string | null;
48+ /** Bulk: repositories imported at the same time. */
49+ concurrency: number;
50+}
51+
52+export const MAX_BULK_REPOS = 1000;
53+const GH_LOGIN = /^[A-Za-z0-9](?:[A-Za-z0-9-]{0,38})$/;
54+const GH_REPO = /^[A-Za-z0-9._-]{1,100}$/;
55+
56+const bad = (msg: string): never => {
57+ throw new HttpError(422, msg);
58+};
59+
60+/**
61+ * Hosts a server-side fetch may reach: HTTPS on a public name. Loopback and
62+ * plain HTTP are allowed only in local development (`dev`), for test doubles.
63+ */
64+export function checkPublicUrl(raw: string, dev: boolean, what = "URL"): URL {
65+ let u: URL;
66+ try {
67+ u = new URL(raw);
68+ } catch {
69+ return bad(`${what} is not a valid URL`);
70+ }
71+ if (u.username || u.password) bad(`${what} must not contain credentials`);
72+ if (dev && (u.protocol === "http:" || u.protocol === "https:")) return u;
73+ if (u.protocol !== "https:") bad(`${what} must use https`);
74+ const host = u.hostname.toLowerCase().replace(/^\[|\]$/g, "");
75+ const singleLabel = !host.includes(".") && !host.includes(":");
76+ if (singleLabel || host === "localhost" || /\.(localhost|local|internal)$/.test(host) || isPrivateAddress(host)) bad(`${what} must be a public host`);
77+ return u;
78+}
79+
80+/** IP literals in loopback, private, link-local, CGNAT or unspecified ranges. */
81+export function isPrivateAddress(host: string): boolean {
82+ const v4 = /^(\d{1,3})\.(\d{1,3})\.(\d{1,3})\.(\d{1,3})$/.exec(host);
83+ if (v4) {
84+ const [a, b] = [Number(v4[1]), Number(v4[2])];
85+ return a === 0 || a === 10 || a === 127 || (a === 169 && b === 254) || (a === 172 && b >= 16 && b <= 31) || (a === 192 && b === 168) || (a === 100 && b >= 64 && b <= 127) || a >= 224;
86+ }
87+ if (host.includes(":")) {
88+ const h = host.toLowerCase();
89+ if (h === "::" || h === "::1" || /^f[cd]/.test(h) || /^fe[89ab]/.test(h)) return true;
90+ // NAT64 well-known prefix (64:ff9b::/96 and 64:ff9b:1::/48) can reach private IPv4 space.
91+ if (h.startsWith("64:ff9b:") || h.startsWith("64:ff9b::")) return true;
92+ // IPv4-mapped (::ffff:a.b.c.d / ::ffff:7f00:1) or -compatible (::7f00:1, ::a.b.c.d): re-check the embedded IPv4.
93+ const embedded = embeddedV4(h);
94+ if (embedded) return isPrivateAddress(embedded);
95+ return false;
96+ }
97+ return false;
98+}
99+
100+/** The IPv4 address embedded in an IPv4-mapped or -compatible IPv6 literal, or null. */
101+function embeddedV4(h: string): string | null {
102+ const dotted = /(?:^|:)((?:\d{1,3}\.){3}\d{1,3})$/.exec(h);
103+ if (dotted) return dotted[1]!;
104+ const m = /^::(?:ffff:)?([0-9a-f]{1,4}):([0-9a-f]{1,4})$/.exec(h);
105+ if (m) {
106+ const hi = parseInt(m[1]!, 16);
107+ const lo = parseInt(m[2]!, 16);
108+ return `${hi >> 8}.${hi & 255}.${lo >> 8}.${lo & 255}`;
109+ }
110+ return null;
111+}
112+
113+/** The REST API base for a GitHub host: github.com → api.github.com, Enterprise → https://host/api/v3. */
114+export function normalizeApiUrl(raw: string | undefined | null, dev: boolean): string {
115+ const s = (raw ?? "").trim();
116+ if (!s) return "https://api.github.com";
117+ const u = checkPublicUrl(s, dev, "github_api_url");
118+ const host = u.host.toLowerCase();
119+ if (host === "github.com" || host === "www.github.com" || host === "api.github.com") return "https://api.github.com";
120+ let p = u.pathname.replace(/\/+$/, "");
121+ if (!p && !dev) p = "/api/v3";
122+ return `${u.protocol}//${u.host}${p}`;
123+}
124+
125+/** The web host that goes with an API base (for clone URLs and links). */
126+export function webHostOf(apiUrl: string): string {
127+ const u = new URL(apiUrl);
128+ return u.host === "api.github.com" ? "github.com" : u.host;
129+}
130+
131+/**
132+ * A repository reference: "owner/name", a github.com or Enterprise URL, or an
133+ * SSH remote. Returns the API base implied by a URL's host, if any.
134+ */
135+export function parseRepoSource(raw: string): { owner: string; name: string; host: string | null } {
136+ let s = raw.trim();
137+ let host: string | null = null;
138+ const ssh = /^git@([^:]+):(.+)$/.exec(s);
139+ if (ssh) {
140+ host = ssh[1]!;
141+ s = ssh[2]!;
142+ } else if (/^https?:\/\//i.test(s) || /^[a-z0-9.-]+\.[a-z]{2,}\//i.test(s)) {
143+ const u = new URL(/^https?:\/\//i.test(s) ? s : `https://${s}`);
144+ host = u.host.toLowerCase();
145+ s = u.pathname;
146+ }
147+ const parts = s.replace(/^\/+|\/+$/g, "").replace(/\.git$/, "").split("/");
148+ if (parts.length < 2 || !parts[0] || !parts[1]) bad(`expected a GitHub repository like owner/name, got "${raw}"`);
149+ const [owner, name] = parts as [string, string];
150+ if (!GH_LOGIN.test(owner) || !GH_REPO.test(name)) bad(`"${raw}" is not a valid GitHub repository`);
151+ return { owner, name, host: host === "github.com" || host === "www.github.com" ? null : host };
152+}
153+
154+function bool(v: unknown, dflt: boolean, key: string): boolean {
155+ if (v === undefined || v === null) return dflt;
156+ if (typeof v !== "boolean") bad(`${key} must be true or false`);
157+ return v as boolean;
158+}
159+
160+export function parseOptions(raw: unknown): ImportOptions {
161+ if (raw === undefined || raw === null) return { ...DEFAULT_OPTIONS };
162+ if (typeof raw !== "object" || Array.isArray(raw)) return bad("options must be an object");
163+ const o = raw as Record<string, unknown>;
164+ const known = new Set(["issues", "releases", "labels", "branch_protection", "variables", "lfs", "secrets"]);
165+ for (const k of Object.keys(o)) if (!known.has(k)) bad(`unknown option "${k}"`);
166+ const secrets = o.secrets ?? DEFAULT_OPTIONS.secrets;
167+ if (secrets !== "bypass" && secrets !== "stop") bad('options.secrets must be "bypass" or "stop"');
168+ return {
169+ issues: bool(o.issues, DEFAULT_OPTIONS.issues, "options.issues"),
170+ releases: bool(o.releases, DEFAULT_OPTIONS.releases, "options.releases"),
171+ labels: bool(o.labels, DEFAULT_OPTIONS.labels, "options.labels"),
172+ protection: bool(o.branch_protection, DEFAULT_OPTIONS.protection, "options.branch_protection"),
173+ variables: bool(o.variables, DEFAULT_OPTIONS.variables, "options.variables"),
174+ lfs: bool(o.lfs, DEFAULT_OPTIONS.lfs, "options.lfs"),
175+ secrets: secrets as ImportOptions["secrets"],
176+ };
177+}
178+
179+/**
180+ * Validate a REST request body (`POST /api/v3/diffinery/imports`):
181+ *
182+ * one repository: { source: "octo/app", owner, name?, visibility?, options?, github_token?, github_api_url? }
183+ * many: { source_owner: "octo", repositories?: [...], exclude_forks?, exclude_archived?, concurrency?, owner, … }
184+ */
185+export function parseImportRequest(body: unknown, o: { dev: boolean }): ImportRequest {
186+ if (!body || typeof body !== "object" || Array.isArray(body)) return bad("request body must be a JSON object");
187+ const b = body as Record<string, unknown>;
188+ const str = (k: string): string | null => {
189+ const v = b[k];
190+ if (v === undefined || v === null || v === "") return null;
191+ if (typeof v !== "string") return bad(`${k} must be a string`);
192+ return v.trim();
193+ };
194+ const single = str("source");
195+ const bulkOwner = str("source_owner");
196+ if (!!single === !!bulkOwner) bad("give either source (one repository) or source_owner (an organization or user)");
197+ const owner = str("owner");
198+ if (!owner) bad("owner (the Diffinery user or organization to import into) is required");
199+ // GitHub "internal" repositories import as private (Diffinery has only public and private).
200+ const rawVisibility = str("visibility") ?? "keep";
201+ const visibility = (rawVisibility === "internal" ? "private" : rawVisibility) as ImportVisibility;
202+ if (!["keep", "private", "public"].includes(visibility)) bad("visibility must be keep, private or public");
203+ const token = str("github_token");
204+ if (token && (token.length > 400 || /\s/.test(token))) bad("github_token is not a valid token");
205+ let apiUrl = str("github_api_url");
206+ let source: string;
207+ let name: string | null = null;
208+ let repositories: string[] | null = null;
209+ if (single) {
210+ const parsed = parseRepoSource(single);
211+ // M3: never send a token to a host taken from the (attacker-influenceable) source URL. When the
212+ // source names an Enterprise host and a token is supplied, the caller must give an explicit,
213+ // matching github_api_url; without a token the source host may imply the API base.
214+ if (parsed.host) {
215+ if (token) {
216+ if (!apiUrl) bad("github_api_url is required when a github_token is given and the source names a GitHub Enterprise host");
217+ if (new URL(normalizeApiUrl(apiUrl, o.dev)).host.toLowerCase() !== parsed.host.toLowerCase()) bad("github_api_url must be on the same host as the source repository");
218+ } else if (!apiUrl) {
219+ apiUrl = `https://${parsed.host}/api/v3`;
220+ }
221+ }
222+ source = `${parsed.owner}/${parsed.name}`;
223+ name = str("name") ?? parsed.name;
224+ if (!isValidName(name)) bad(`"${name}" is not a valid repository name: use letters, numbers, '.', '-' and '_'`);
225+ if (b.repositories !== undefined) bad("repositories applies only to source_owner imports");
226+ } else {
227+ if (!GH_LOGIN.test(bulkOwner!)) bad(`"${bulkOwner}" is not a valid GitHub organization or user`);
228+ source = bulkOwner!;
229+ if (b.name !== undefined) bad("name applies only to single repository imports");
230+ if (b.repositories !== undefined && b.repositories !== null) {
231+ if (!Array.isArray(b.repositories) || b.repositories.some((r) => typeof r !== "string" || !GH_REPO.test(r))) bad("repositories must be a list of repository names");
232+ repositories = [...new Set(b.repositories as string[])];
233+ if (!repositories.length) bad("repositories is empty; omit it to import every repository");
234+ if (repositories.length > MAX_BULK_REPOS) bad(`at most ${MAX_BULK_REPOS} repositories per import`);
235+ }
236+ }
237+ const concurrency = b.concurrency === undefined || b.concurrency === null ? 3 : Number(b.concurrency);
238+ if (!Number.isInteger(concurrency) || concurrency < 1 || concurrency > 8) bad("concurrency must be between 1 and 8");
239+ return {
240+ mode: single ? "repo" : "bulk",
241+ source,
242+ repositories,
243+ excludeForks: bool(b.exclude_forks, false, "exclude_forks"),
244+ excludeArchived: bool(b.exclude_archived, false, "exclude_archived"),
245+ owner: owner!,
246+ name,
247+ visibility,
248+ options: parseOptions(b.options),
249+ apiUrl: normalizeApiUrl(apiUrl, o.dev),
250+ token,
251+ concurrency,
252+ };
253+}
254+
255+/**
256+ * A safe asset filename: the final path segment with path-dangerous characters
257+ * removed. Rejects "", ".", "..", any ".git" variant, and anything with a
258+ * backslash or NUL, so nothing server-side ever writes a traversing name.
259+ */
260+export function safeAssetName(raw: string | null | undefined): string | null {
261+ if (!raw) return null;
262+ if (raw.includes("\0")) return null;
263+ const base = raw.split(/[\\/]/).pop() ?? "";
264+ // Reject dangerous names on the raw final segment, before cleaning strips the leading dot off ".git".
265+ if ([".", "..", ".git", ".git.", "git~1"].includes(base.toLowerCase())) return null;
266+ const cleaned = base.replace(/[^A-Za-z0-9._+@~-]+/g, ".").replace(/^\.+|\.+$/g, "");
267+ if (!cleaned) return null;
268+ if ([".", "..", ".git"].includes(cleaned.toLowerCase())) return null;
269+ return cleaned.slice(0, 255);
270+}
271+
272+/** Diffinery visibility for a GitHub repository: only public or private (GitHub "internal" becomes private). */
273+export function visibilityFor(v: ImportVisibility, gh: { private?: boolean; visibility?: string }): "private" | "public" {
274+ if (v === "public") return "public";
275+ if (v === "private") return "private";
276+ // "keep": public stays public; private and internal both become private.
277+ return gh.visibility === "public" || (gh.visibility === undefined && gh.private === false) ? "public" : "private";
278+}
addedapps/diffinery/src/imports/service.ts+1595−0View file
@@ -1,0 +1,1595 @@
1+import { NonRetryableError } from "cloudflare:workflows";
2+import type { ImportGitResult, ImportProgress, ImportSecretFinding, ImportSpec } from "@diffinery/shared";
3+import type { Env } from "../env.js";
4+import { decryptSecret, encryptSecret } from "../lib/crypto.js";
5+import { Directory, type Account, type Repo, type TokenScope } from "../lib/directory.js";
6+import { atLeast } from "../repo/types.js";
7+import { HttpError, newId, now } from "../lib/util.js";
8+import { GitHubClient, GitHubError, RateLimited, listPath } from "./github.js";
9+import {
10+ People,
11+ mapIssue,
12+ mapIssueComment,
13+ mapMilestone,
14+ mapProtection,
15+ mapPull,
16+ mapReview,
17+ mapReviewComment,
18+ type GhComment,
19+ type GhIssue,
20+ type GhLabel,
21+ type GhMilestone,
22+ type GhProtection,
23+ type GhPull,
24+ type GhReview,
25+ type GhReviewComment,
26+ type GhUser,
27+ type Link,
28+} from "./mapping.js";
29+import { MAX_BULK_REPOS, parseImportRequest, safeAssetName, visibilityFor, webHostOf, normalizeApiUrl, type ImportOptions, type ImportVisibility } from "./options.js";
30+
31+/**
32+ * GitHub imports: the records in D1 (`imports`), the steps the import workflow
33+ * runs (see workflow.ts), and the endpoints of the Git transfer agent.
34+ *
35+ * An import creates the repository, has the import agent copy Git data in a
36+ * container, then copies issues, pull requests, comments, reviews, labels,
37+ * milestones, branch protection and variables through the GitHub REST API.
38+ * Every write is an upsert keyed by GitHub ids, so a retried step or a retried
39+ * import never duplicates anything, and nothing an import writes fires
40+ * workflows, webhooks or notifications.
41+ */
42+
43+export type ImportStatus = "draft" | "queued" | "running" | "succeeded" | "failed" | "cancelled";
44+
45+export const REPO_STEPS = [
46+ ["repository", "Create the repository"],
47+ ["git", "Copy branches, tags and history"],
48+ ["labels", "Labels and milestones"],
49+ ["issues", "Issues and pull requests"],
50+ ["comments", "Comments"],
51+ ["reviews", "Reviews"],
52+ ["review_comments", "Review comments"],
53+ ["protection", "Branch protection"],
54+ ["variables", "Variables and secret names"],
55+ ["releases", "Releases"],
56+ ["finalize", "Finish"],
57+] as const;
58+
59+export const BULK_STEPS = [
60+ ["organization", "Organization variables and secret names"],
61+ ["repositories", "Repositories"],
62+ ["finalize", "Finish"],
63+] as const;
64+
65+/** Steps that page through GitHub; the workflow calls them until they report done. */
66+export const PAGED_STEPS = ["labels", "issues", "comments", "reviews", "review_comments", "protection", "variables", "releases"] as const;
67+export type PagedStep = (typeof PAGED_STEPS)[number];
68+export type StepCursor = string | number | { phase: string; next: string } | null;
69+
70+export interface StepState {
71+ status: "pending" | "running" | "done" | "skipped" | "failed";
72+ message?: string;
73+ count?: number;
74+ startedAt?: string;
75+ finishedAt?: string;
76+}
77+
78+export interface ImportRow {
79+ id: string;
80+ parent_id: string | null;
81+ kind: "repo" | "bulk";
82+ status: ImportStatus;
83+ created_by: string;
84+ api_url: string;
85+ source: string;
86+ target_owner_id: string;
87+ target_name: string | null;
88+ repo_id: string | null;
89+ request: string;
90+ steps: string;
91+ git: string | null;
92+ summary: string;
93+ warnings: string;
94+ error: string | null;
95+ token_ciphertext: string | null;
96+ push_token_id: string | null;
97+ attempt: number;
98+ created_at: string;
99+ updated_at: string;
100+ started_at: string | null;
101+ finished_at: string | null;
102+}
103+
104+export interface Candidate {
105+ name: string;
106+ private: boolean;
107+ fork: boolean;
108+ archived: boolean;
109+ description: string;
110+}
111+
112+export interface StoredRequest {
113+ visibility: ImportVisibility;
114+ options: ImportOptions;
115+ /** Bulk: GitHub account type of the source ("Organization" or "User"). */
116+ sourceType?: string;
117+ repositories: string[] | null;
118+ excludeForks: boolean;
119+ excludeArchived: boolean;
120+ concurrency: number;
121+ /** Token scopes of the person or agent who started the import; the push token gets the same. */
122+ scopes: TokenScope[];
123+ /** Drafts of bulk imports: the repositories to choose from. */
124+ candidates?: Candidate[];
125+}
126+
127+export interface GitState {
128+ phase?: string;
129+ message?: string;
130+ chunks?: number;
131+ bytes?: number;
132+ at?: string;
133+ tries?: number;
134+ result?: ImportGitResult;
135+}
136+
137+export interface ImportSummary {
138+ source?: {
139+ htmlUrl: string;
140+ cloneUrl: string;
141+ defaultBranch: string;
142+ visibility: string;
143+ archived: boolean;
144+ description: string;
145+ homepage: string;
146+ topics: string[];
147+ };
148+ git?: Omit<ImportGitResult, "ok" | "error" | "secretFindings" | "warnings">;
149+ counts?: { issues: number; pulls: number; comments: number; reviews: number; labels: number; milestones: number };
150+ pullRefs?: { created: number; missing: number[] };
151+ protections?: string[];
152+ variables?: string[];
153+ orgVariables?: string[];
154+ secretNames?: string[];
155+ orgSecretNames?: string[];
156+ releases?: { imported: number; assets: number; skippedAssets: number };
157+ environments?: { imported: number; variables: number; secretNames: string[] };
158+ findings?: ImportSecretFinding[];
159+ people?: { linked: string[]; imported: string[] };
160+ children?: { total: number; succeeded: number; failed: number; cancelled: number; open: number };
161+}
162+
163+/** Default size bound for one push: well under the 100 MB request cap and the 128 MB of a repository's Durable Object. */
164+export const DEFAULT_CHUNK_BYTES = 32 * 1024 * 1024;
165+// Larger batches per workflow step keep the total number of workflow steps well under the platform cap.
166+const PAGES_PER_STEP = 10;
167+const PULLS_PER_STEP = 30;
168+/** M5 import volume limits (per account and per organization). */
169+const MAX_ACTIVE_BULK = 1;
170+const MAX_ACTIVE_REPO = 3;
171+/** Largest source repository an import will take, in KB (GitHub's `size`); overridable with IMPORT_MAX_REPO_KB. */
172+const DEFAULT_MAX_REPO_KB = 10 * 1024 * 1024;
173+
174+const fatal = (message: string) => new NonRetryableError(message);
175+const errMsg = (e: unknown) => (e instanceof Error ? e.message.replace(/^\d{3}: /, "") : String(e));
176+
177+/** A valid Actions variable name: GitHub's rules (letters, digits, underscore, not starting with a digit) and never the reserved GITHUB_ prefix. */
178+const validVarName = (name: string): boolean => /^[A-Za-z_][A-Za-z0-9_]*$/.test(name) && !/^GITHUB_/i.test(name) && name.length <= 256;
179+
180+/**
181+ * Whether a Durable Object RPC failed because the method does not exist on this
182+ * build of RepoDO (releases and environments are delivered by a separate
183+ * branch). A stub proxy reports every property as callable, so a missing method
184+ * can only be told apart by the error from calling it.
185+ */
186+const methodMissing = (e: unknown): boolean => /not (a )?(implement|function)|does not implement|no such method|is not a function/i.test(e instanceof Error ? e.message : String(e));
187+
188+/** A GitHub release and its assets, as returned by `GET /repos/{owner}/{repo}/releases`. */
189+interface GhRelease {
190+ tag_name: string;
191+ target_commitish?: string;
192+ name?: string | null;
193+ body?: string | null;
194+ draft?: boolean;
195+ prerelease?: boolean;
196+ created_at: string;
197+ published_at?: string | null;
198+ author?: GhUser | null;
199+ assets?: GhReleaseAsset[];
200+}
201+interface GhReleaseAsset {
202+ /** The API URL; downloading needs `Accept: application/octet-stream` and redirects to a GitHub asset host. */
203+ url: string;
204+ name: string;
205+ label?: string | null;
206+ content_type?: string;
207+ size?: number;
208+ download_count?: number;
209+ created_at?: string;
210+ updated_at?: string;
211+ uploader?: GhUser | null;
212+}
213+
214+/** A GitHub deployment environment, from `GET /repos/{owner}/{repo}/environments`. */
215+interface GhEnvironment {
216+ name: string;
217+ deployment_branch_policy?: { protected_branches?: boolean; custom_branch_policies?: boolean } | null;
218+}
219+
220+/** The import's own writer identity for RepoDO calls that require a caller. */
221+interface ImportCaller {
222+ id: string;
223+ login: string;
224+ kind: "user" | "agent" | "system";
225+ role: "admin";
226+ importId: string;
227+}
228+
229+/**
230+ * The release-import surface of RepoDO (delivered by a separate branch). Named
231+ * structurally so this module compiles and runs whether or not that branch is
232+ * present; when it is absent the release step detects the missing methods and
233+ * records that releases were not imported.
234+ */
235+interface ReleaseImporter {
236+ importRelease(input: { tag_name: string; target_commitish?: string; name?: string | null; body?: string | null; draft?: boolean; prerelease?: boolean; created_at: string; published_at?: string | null; author: { id?: string; login: string; kind?: string }; latest?: boolean }, caller: ImportCaller): Promise<{ id: number }> | { id: number };
237+ importReleaseAsset(
238+ releaseId: number,
239+ meta: { name: string; label?: string; content_type?: string; download_count?: number; created_at?: string; updated_at?: string; uploader?: { id?: string; login: string; kind?: string } },
240+ source: { r2Key: string } | { url: string; headers?: Record<string, string> },
241+ caller: ImportCaller,
242+ ): Promise<unknown>;
243+}
244+
245+/**
246+ * The environment-import surface of RepoDO (also from the separate branch).
247+ * Secret *values* are never copied (only their names are listed), matching the
248+ * repository-level policy: an admin re-enters them by hand.
249+ */
250+interface EnvImporter {
251+ upsertEnvironment(name: string, input: { deployment_branch_policy?: { protected_branches?: boolean; custom_branch_policies?: boolean } | null }, caller: ImportCaller): unknown;
252+ addEnvironmentBranchPattern(name: string, pattern: { name?: string; type?: string }, caller: ImportCaller): unknown;
253+ setEnvironmentVariable(envName: string, name: string, value: string | null, caller: ImportCaller): unknown;
254+}
255+
256+const parse = <T>(s: string | null | undefined, fallback: T): T => {
257+ if (!s) return fallback;
258+ try {
259+ return JSON.parse(s) as T;
260+ } catch {
261+ return fallback;
262+ }
263+};
264+
265+/**
266+ * Whether `importId`'s push token may push to `repoId` in import mode right now. The token must be
267+ * the import's *current* push token (M4): a token from an earlier attempt, though still a valid
268+ * import token, has a different id and is refused, so a stale container can never push.
269+ */
270+export async function importPushAllowed(env: Env, importId: string, repoId: string, tokenId: string): Promise<boolean> {
271+ const r = await env.DB.prepare("SELECT status, repo_id, push_token_id FROM imports WHERE id = ?").bind(importId).first<{ status: string; repo_id: string | null; push_token_id: string | null }>();
272+ return r?.status === "running" && r.repo_id === repoId && r.push_token_id === tokenId;
273+}
274+
275+interface Ctx {
276+ row: ImportRow;
277+ req: StoredRequest;
278+ gh: GitHubClient;
279+ /** The GitHub token (decrypted), needed to authenticate release-asset downloads. */
280+ token: string | null;
281+ people: People;
282+ summary: ImportSummary;
283+ steps: Record<string, StepState>;
284+ repo: Repo | null;
285+ stub: DurableObjectStub<import("../repo/RepoDO.js").RepoDO> | null;
286+}
287+
288+interface StepOutcome {
289+ done: boolean;
290+ cursor?: StepCursor;
291+ /** Items written in this call (added to the step's count). */
292+ added?: number;
293+ message?: string;
294+ skipped?: boolean;
295+}
296+
297+export class ImportService {
298+ readonly dir: Directory;
299+
300+ constructor(private readonly env: Env) {
301+ this.dir = new Directory(env);
302+ }
303+
304+ /** Local development: plain HTTP and loopback GitHub doubles are allowed. */
305+ get dev(): boolean {
306+ return !!this.env.DEV_AUTH && !this.env.ACCESS_AUD;
307+ }
308+
309+ private get db() {
310+ return this.env.DB;
311+ }
312+
313+ // ---------------------------------------------------------------- records
314+
315+ async row(id: string): Promise<ImportRow | null> {
316+ return this.db.prepare("SELECT * FROM imports WHERE id = ?").bind(id).first<ImportRow>();
317+ }
318+
319+ private async mustRow(id: string): Promise<ImportRow> {
320+ const r = await this.row(id);
321+ if (!r) throw fatal(`import ${id} not found`);
322+ return r;
323+ }
324+
325+ async list(accountId: string, limit = 100): Promise<ImportRow[]> {
326+ return (await this.db.prepare("SELECT * FROM imports WHERE created_by = ? AND parent_id IS NULL ORDER BY created_at DESC LIMIT ?").bind(accountId, limit).all<ImportRow>()).results;
327+ }
328+
329+ async children(parentId: string): Promise<ImportRow[]> {
330+ return (await this.db.prepare("SELECT * FROM imports WHERE parent_id = ? ORDER BY created_at, source").bind(parentId).all<ImportRow>()).results;
331+ }
332+
333+ /** People who may see an import: who started it, and admins of the account it imports into. */
334+ async visibleTo(row: ImportRow, actor: Account): Promise<boolean> {
335+ if (row.created_by === actor.id || actor.site_admin) return true;
336+ if (actor.kind !== "user") return false;
337+ const owner = await this.dir.account(row.target_owner_id);
338+ return !!owner && owner.kind !== "agent" && (await this.canImportInto(actor, owner));
339+ }
340+
341+ /**
342+ * People import into their own account or organizations they own or administer.
343+ * Agents import only into organizations they belong to, or whose owner or admin owns them.
344+ */
345+ async canImportInto(actor: Account, owner: Account): Promise<boolean> {
346+ if (actor.status !== "active" || owner.kind === "agent") return false;
347+ if (actor.kind === "agent") {
348+ if (owner.kind !== "org" || !actor.owner_id) return false;
349+ if (actor.owner_id === owner.id) return true;
350+ const human = await this.dir.account(actor.owner_id);
351+ if (!human || human.kind !== "user" || human.status !== "active") return false;
352+ const role = await this.dir.orgRole(owner.id, human.id);
353+ return role === "owner" || role === "admin";
354+ }
355+ if (actor.kind !== "user") return false;
356+ if (actor.site_admin) return true;
357+ if (owner.kind === "user") return owner.id === actor.id;
358+ const role = await this.dir.orgRole(owner.id, actor.id);
359+ return role === "owner" || role === "admin";
360+ }
361+
362+ private async setStep(id: string, name: string, patch: Partial<StepState>): Promise<void> {
363+ const clean = Object.fromEntries(Object.entries(patch).filter(([, v]) => v !== undefined));
364+ await this.db
365+ .prepare("UPDATE imports SET steps = json_set(steps, '$.' || ?, json_patch(COALESCE(json_extract(steps, '$.' || ?), '{}'), json(?))), updated_at = ? WHERE id = ?")
366+ .bind(name, name, JSON.stringify(clean), now(), id)
367+ .run();
368+ }
369+
370+ private async saveSummary(id: string, summary: ImportSummary): Promise<void> {
371+ await this.db.prepare("UPDATE imports SET summary = ?, updated_at = ? WHERE id = ?").bind(JSON.stringify(summary), now(), id).run();
372+ }
373+
374+ async warn(id: string, message: string): Promise<void> {
375+ await this.db.prepare("UPDATE imports SET warnings = json_insert(warnings, '$[#]', ?), updated_at = ? WHERE id = ? AND json_array_length(warnings) < 500").bind(message.slice(0, 1000), now(), id).run();
376+ }
377+
378+ private async audit(row: ImportRow, action: string, detail: Record<string, unknown>): Promise<void> {
379+ const actor = await this.dir.account(row.created_by);
380+ await this.dir.audit({ actorId: row.created_by, actorLogin: actor?.login, action, target: row.source, detail: { import: row.id, ...detail } });
381+ }
382+
383+ // ---------------------------------------------------------------- starting imports
384+
385+ /**
386+ * Validate and record an import, then start its workflow. With `draft`, a bulk
387+ * import without a selection is recorded with its candidate repositories and
388+ * waits for `select` (the web form's second page).
389+ */
390+ async create(actor: Account, scopes: TokenScope[], body: unknown, o: { draft?: boolean } = {}): Promise<ImportRow> {
391+ if (actor.kind === "org") throw new HttpError(403, "organizations cannot start imports");
392+ if (!scopes.includes("write") && !scopes.includes("admin")) throw new HttpError(403, "starting an import needs a token with write or admin scope");
393+ const req = parseImportRequest(body, { dev: this.dev });
394+ const owner = await this.dir.accountByLogin(req.owner);
395+ if (!owner || owner.kind === "agent") throw new HttpError(422, `no Diffinery user or organization named ${req.owner}`);
396+ if (!(await this.canImportInto(actor, owner))) {
397+ throw new HttpError(403, actor.kind === "agent" ? "agents can import only into organizations they or their owner administer" : `you can import only into your own account or organizations you administer, not ${owner.login}`);
398+ }
399+ // An agent may import only if its policy lets it write (H1): an import creates and fills a repository.
400+ if (actor.kind === "agent" && !atLeast(this.dir.agentMaxRole(actor), "write")) throw new HttpError(403, "this agent's policy does not allow writing; an import needs at least write");
401+ // Agents can never bypass push protection: an import they start imports secret-bearing
402+ // history only if it is clean, never with an admin bypass.
403+ if (actor.kind === "agent") req.options.secrets = "stop";
404+ // M5: cap how many imports one account or organization can run at once.
405+ await this.assertImportQuota(actor.id, owner.id, req.mode);
406+ const gh = new GitHubClient(req.apiUrl, req.token);
407+ const stored: StoredRequest = {
408+ visibility: req.visibility,
409+ options: req.options,
410+ repositories: req.repositories,
411+ excludeForks: req.excludeForks,
412+ excludeArchived: req.excludeArchived,
413+ concurrency: req.concurrency,
414+ scopes: scopes.includes("admin") ? ["admin"] : ["write"],
415+ };
416+ const id = newId();
417+ const t = now();
418+ const ciphertext = req.token ? await encryptSecret(this.env.MASTER_KEY, `import:${id}`, req.token) : null;
419+ const insert = (r: Partial<ImportRow> & { id: string; kind: ImportRow["kind"]; status: ImportStatus; source: string }) =>
420+ this.db
421+ .prepare(
422+ "INSERT INTO imports (id, parent_id, kind, status, created_by, api_url, source, target_owner_id, target_name, request, steps, summary, warnings, error, token_ciphertext, attempt, created_at, updated_at) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, '{}', '{}', '[]', ?, ?, 1, ?, ?)",
423+ )
424+ .bind(r.id, r.parent_id ?? null, r.kind, r.status, actor.id, req.apiUrl, r.source, owner.id, r.target_name ?? null, r.request ?? JSON.stringify(stored), r.error ?? null, r.token_ciphertext ?? null, t, t);
425+
426+ if (req.mode === "repo") {
427+ const gr = await this.sourceRepo(gh, req.source);
428+ this.assertRepoSize(gr.size);
429+ if (await this.dir.repo(owner.login, req.name!)) throw new HttpError(422, `${owner.login}/${req.name} already exists; choose another name`);
430+ await insert({ id, kind: "repo", status: "queued", source: gr.full_name, target_name: req.name, token_ciphertext: ciphertext ?? undefined }).run();
431+ } else {
432+ const { type, repos } = await this.sourceRepos(gh, req.source);
433+ stored.sourceType = type;
434+ let candidates = repos.filter((r) => !(req.excludeForks && r.fork) && !(req.excludeArchived && r.archived));
435+ if (req.repositories) {
436+ const byName = new Map(repos.map((r) => [r.name.toLowerCase(), r]));
437+ const unknown = req.repositories.filter((n) => !byName.has(n.toLowerCase()));
438+ if (unknown.length) throw new HttpError(422, `not found in ${req.source}: ${unknown.slice(0, 10).join(", ")}`);
439+ candidates = req.repositories.map((n) => byName.get(n.toLowerCase())!);
440+ }
441+ if (!candidates.length) throw new HttpError(422, `${req.source} has no repositories to import`);
442+ if (o.draft && !req.repositories) {
443+ await insert({ id, kind: "bulk", status: "draft", source: req.source, request: JSON.stringify({ ...stored, candidates }), token_ciphertext: ciphertext ?? undefined }).run();
444+ return (await this.row(id))!;
445+ }
446+ await insert({ id, kind: "bulk", status: "queued", source: req.source, request: JSON.stringify(stored), token_ciphertext: ciphertext ?? undefined }).run();
447+ await this.createChildren((await this.row(id))!, owner, candidates.map((c) => c.name), req.token);
448+ }
449+ const row = (await this.row(id))!;
450+ await this.audit(row, "import.create", { kind: row.kind, owner: owner.login, name: row.target_name, options: { ...req.options } });
451+ await this.launch(id, 1);
452+ return row;
453+ }
454+
455+ /** Refuse a repository whose GitHub size (KB) is over the configured ceiling (default 10 GB). */
456+ private assertRepoSize(sizeKb: number | undefined): void {
457+ const maxKb = Number(this.env.IMPORT_MAX_REPO_KB) || DEFAULT_MAX_REPO_KB;
458+ if (typeof sizeKb === "number" && sizeKb > maxKb) {
459+ throw new HttpError(422, `the repository is about ${Math.round(sizeKb / 1024 / 1024)} GB, over the ${Math.round(maxKb / 1024 / 1024)} GB import limit; ask an administrator to raise IMPORT_MAX_REPO_KB or split the repository`);
460+ }
461+ }
462+
463+ /** M5: at most one active bulk import and three active repository imports per account and per organization. */
464+ private async assertImportQuota(accountId: string, ownerId: string, mode: "repo" | "bulk"): Promise<void> {
465+ const count = async (col: "created_by" | "target_owner_id", id: string): Promise<number> => {
466+ if (mode === "bulk") return (await this.db.prepare(`SELECT count(*) AS n FROM imports WHERE ${col} = ? AND kind = 'bulk' AND status IN ('draft','queued','running')`).bind(id).first<{ n: number }>())?.n ?? 0;
467+ return (await this.db.prepare(`SELECT count(*) AS n FROM imports WHERE ${col} = ? AND kind = 'repo' AND parent_id IS NULL AND status IN ('queued','running')`).bind(id).first<{ n: number }>())?.n ?? 0;
468+ };
469+ const limit = mode === "bulk" ? MAX_ACTIVE_BULK : MAX_ACTIVE_REPO;
470+ if ((await count("created_by", accountId)) >= limit || (await count("target_owner_id", ownerId)) >= limit) {
471+ throw new HttpError(429, mode === "bulk" ? `an organization import is already in progress (at most ${MAX_ACTIVE_BULK} at a time); wait for it to finish or cancel it` : `too many repository imports are already running (at most ${MAX_ACTIVE_REPO} at a time); wait for one to finish`);
472+ }
473+ }
474+
475+ /** Bulk drafts: the repositories to import were chosen; start the import. */
476+ async select(id: string, actor: Account, names: string[]): Promise<ImportRow> {
477+ const row = await this.row(id);
478+ if (!row || row.created_by !== actor.id || row.kind !== "bulk") throw new HttpError(404, "import not found");
479+ if (row.status !== "draft") throw new HttpError(409, "this import has already started");
480+ const req = parse<StoredRequest>(row.request, {} as StoredRequest);
481+ const allowed = new Map((req.candidates ?? []).map((c) => [c.name.toLowerCase(), c.name]));
482+ const chosen = [...new Set(names.map((n) => allowed.get(n.toLowerCase())).filter((n): n is string => !!n))];
483+ if (!chosen.length) throw new HttpError(422, "choose at least one repository");
484+ if (chosen.length > MAX_BULK_REPOS) throw new HttpError(422, `at most ${MAX_BULK_REPOS} repositories per import`);
485+ const owner = (await this.dir.account(row.target_owner_id))!;
486+ if (!(await this.canImportInto(actor, owner))) throw new HttpError(403, `you can no longer import into ${owner.login}`);
487+ const token = row.token_ciphertext ? await decryptSecret(this.env.MASTER_KEY, `import:${id}`, row.token_ciphertext) : null;
488+ const { candidates: _c, ...rest } = req;
489+ await this.db.prepare("UPDATE imports SET status = 'queued', request = ?, updated_at = ? WHERE id = ?").bind(JSON.stringify({ ...rest, repositories: chosen }), now(), id).run();
490+ await this.createChildren((await this.row(id))!, owner, chosen, token);
491+ await this.audit(row, "import.create", { kind: "bulk", owner: owner.login, repositories: chosen.length });
492+ await this.launch(id, row.attempt);
493+ return (await this.row(id))!;
494+ }
495+
496+ private async createChildren(parent: ImportRow, owner: Account, names: string[], token: string | null): Promise<void> {
497+ const existing = new Set((await this.db.prepare("SELECT name FROM repos WHERE owner_id = ?").bind(owner.id).all<{ name: string }>()).results.map((r) => r.name.toLowerCase()));
498+ const t = now();
499+ const stmts: D1PreparedStatement[] = [];
500+ for (const name of names) {
501+ const id = newId();
502+ const clash = existing.has(name.toLowerCase());
503+ stmts.push(
504+ this.db
505+ .prepare(
506+ "INSERT INTO imports (id, parent_id, kind, status, created_by, api_url, source, target_owner_id, target_name, request, steps, summary, warnings, error, token_ciphertext, attempt, created_at, updated_at, finished_at) VALUES (?, ?, 'repo', ?, ?, ?, ?, ?, ?, ?, '{}', '{}', '[]', ?, ?, 1, ?, ?, ?)",
507+ )
508+ .bind(
509+ id,
510+ parent.id,
511+ clash ? "failed" : "queued",
512+ parent.created_by,
513+ parent.api_url,
514+ `${parent.source}/${name}`,
515+ owner.id,
516+ name,
517+ parent.request,
518+ clash ? `${owner.login}/${name} already exists` : null,
519+ token && !clash ? await encryptSecret(this.env.MASTER_KEY, `import:${id}`, token) : null,
520+ t,
521+ t,
522+ clash ? t : null,
523+ ),
524+ );
525+ }
526+ for (let i = 0; i < stmts.length; i += 50) await this.db.batch(stmts.slice(i, i + 50));
527+ }
528+
529+ private async launch(id: string, attempt: number): Promise<void> {
530+ await this.env.IMPORT_WORKFLOW.create({ id: `${id}-${attempt}`, params: { importId: id } });
531+ }
532+
533+ /** The GitHub repository, with errors a person can act on. */
534+ private async sourceRepo(gh: GitHubClient, fullName: string): Promise<GhRepo> {
535+ try {
536+ return await gh.get<GhRepo>(`repos/${fullName}`);
537+ } catch (e) {
538+ if (e instanceof GitHubError && (e.status === 404 || e.status === 403)) throw new HttpError(422, `GitHub repository ${fullName} was not found or the token cannot read it`);
539+ if (e instanceof GitHubError && e.status === 401) throw new HttpError(422, "GitHub rejected the token (expired or revoked?)");
540+ if (e instanceof RateLimited) throw new HttpError(429, e.message);
541+ throw new HttpError(502, `could not reach GitHub: ${errMsg(e)}`);
542+ }
543+ }
544+
545+ /** Every repository of a GitHub organization or user (private ones too when the token can see them). */
546+ private async sourceRepos(gh: GitHubClient, owner: string): Promise<{ type: string; repos: Candidate[] }> {
547+ try {
548+ const account = await gh.get<{ login: string; type: string }>(`users/${owner}`);
549+ let path: string;
550+ if (account.type === "Organization") path = listPath(`orgs/${account.login}/repos`, { type: "all" });
551+ else {
552+ const me = await gh.get<{ login: string }>("user").catch(() => null);
553+ path = me?.login.toLowerCase() === account.login.toLowerCase() ? listPath("user/repos", { affiliation: "owner", visibility: "all" }) : listPath(`users/${account.login}/repos`, { type: "owner" });
554+ }
555+ const repos = await gh.all<GhRepo>(path, MAX_BULK_REPOS);
556+ return {
557+ type: account.type,
558+ repos: repos
559+ .filter((r) => r.owner?.login?.toLowerCase() === account.login.toLowerCase())
560+ .map((r) => ({ name: r.name, private: !!r.private, fork: !!r.fork, archived: !!r.archived, description: r.description ?? "" }))
561+ .sort((a, b) => a.name.localeCompare(b.name)),
562+ };
563+ } catch (e) {
564+ if (e instanceof GitHubError && e.status === 404) throw new HttpError(422, `no GitHub organization or user named ${owner}`);
565+ if (e instanceof GitHubError && e.status === 401) throw new HttpError(422, "GitHub rejected the token (expired or revoked?)");
566+ if (e instanceof RateLimited) throw new HttpError(429, e.message);
567+ if (e instanceof HttpError) throw e;
568+ throw new HttpError(502, `could not reach GitHub: ${errMsg(e)}`);
569+ }
570+ }
571+
572+ // ---------------------------------------------------------------- cancel and retry
573+
574+ async cancel(id: string, actor: Account): Promise<ImportRow> {
575+ const row = await this.row(id);
576+ if (!row || !(await this.visibleTo(row, actor))) throw new HttpError(404, "import not found");
577+ if (!["draft", "queued", "running"].includes(row.status)) throw new HttpError(409, `the import already ${row.status}`);
578+ const targets = row.kind === "bulk" ? [row, ...(await this.children(id)).filter((c) => ["queued", "running"].includes(c.status))] : [row];
579+ for (const r of targets) {
580+ await this.close(r, "cancelled", `cancelled by ${actor.login}`);
581+ await this.stopInstance(r);
582+ }
583+ // One repository of a bulk import: the next one starts in its place.
584+ if (row.parent_id) await this.afterChild(row.id);
585+ await this.audit(row, "import.cancel", { by: actor.login });
586+ return (await this.row(id))!;
587+ }
588+
589+ /** Run a failed or cancelled import again. Everything is an upsert, so finished work is not duplicated. */
590+ async retry(id: string, actor: Account, token: string | null): Promise<ImportRow> {
591+ const row = await this.row(id);
592+ // Retry is creator-only (Low 7): even an admin of the target cannot resume someone else's import.
593+ if (!row || row.created_by !== actor.id) throw new HttpError(404, "import not found");
594+ if (row.parent_id) throw new HttpError(422, "retry the organization import this repository belongs to");
595+ if (row.status !== "failed" && row.status !== "cancelled") throw new HttpError(409, "only failed or cancelled imports can be retried");
596+ if (token !== null && (token.length > 400 || /\s/.test(token))) throw new HttpError(422, "github_token is not a valid token");
597+ // H2: once the repository exists, retry replays writes into it. An agent must not do that on its
598+ // own; a person has to take over the retry (and provide a token for a private source).
599+ if (actor.kind === "agent" && row.repo_id) throw new HttpError(403, "an agent cannot retry an import after the repository was created; a person must retry it");
600+ const owner = await this.dir.account(row.target_owner_id);
601+ if (!owner || !(await this.canImportInto(actor, owner))) throw new HttpError(403, "you cannot import into this account");
602+ const attempt = row.attempt + 1;
603+ const t = now();
604+ const reset = async (r: ImportRow, n: number) => {
605+ const ciphertext = token ? await encryptSecret(this.env.MASTER_KEY, `import:${r.id}`, token) : r.token_ciphertext;
606+ await this.db
607+ .prepare("UPDATE imports SET status = 'queued', attempt = ?, error = NULL, steps = '{}', warnings = '[]', token_ciphertext = ?, started_at = NULL, finished_at = NULL, updated_at = ? WHERE id = ?")
608+ .bind(n, ciphertext, t, r.id)
609+ .run();
610+ };
611+ if (row.kind === "bulk") {
612+ for (const c of await this.children(id)) {
613+ if (c.status === "failed" || c.status === "cancelled") {
614+ if (c.error?.endsWith("already exists") && !c.repo_id) continue;
615+ await reset(c, c.attempt + 1);
616+ }
617+ }
618+ }
619+ await reset(row, attempt);
620+ await this.audit(row, "import.retry", { by: actor.login, attempt });
621+ await this.launch(id, attempt);
622+ return (await this.row(id))!;
623+ }
624+
625+ private async stopInstance(row: ImportRow): Promise<void> {
626+ try {
627+ await (await this.env.IMPORT_WORKFLOW.get(`${row.id}-${row.attempt}`)).terminate();
628+ } catch {
629+ // not running
630+ }
631+ await this.stopContainer(row);
632+ }
633+
634+ private async stopContainer(row: ImportRow): Promise<void> {
635+ const git = parse<GitState>(row.git, {});
636+ if (!this.env.IMPORTER || !git.tries) return;
637+ // Stop the container of every try of this attempt, not just the latest, so no leaked container keeps pushing.
638+ for (let t = 1; t <= git.tries; t++) {
639+ await this.env.IMPORTER.get(this.env.IMPORTER.idFromName(`${row.id}-${row.attempt}-${t}`)).stopImport().catch(() => {});
640+ }
641+ }
642+
643+ /** Finish an import: final status, and its credentials are gone (every push token it minted revoked, the GitHub token deleted). */
644+ private async close(row: ImportRow, status: "succeeded" | "failed" | "cancelled", error: string | null): Promise<void> {
645+ // M4: revoke every token this import ever minted, not just the current one, so a stale token can never push.
646+ await this.dir.revokeTokensByImport(row.id);
647+ // A failed or cancelled import gives the repository back: release the lock and drop its in-flight refs.
648+ if ((status === "failed" || status === "cancelled") && row.repo_id) {
649+ try {
650+ const stub = this.env.REPO.get(this.env.REPO.idFromName(row.repo_id));
651+ await stub.importDropTempRefs(row.id);
652+ await stub.importEnd(row.id);
653+ } catch (e) {
654+ console.error("releasing import lock failed", row.id, e);
655+ }
656+ }
657+ const t = now();
658+ await this.db
659+ .prepare("UPDATE imports SET status = ?, error = ?, token_ciphertext = NULL, finished_at = ?, updated_at = ? WHERE id = ? AND status IN ('draft', 'queued', 'running')")
660+ .bind(status, error ? error.slice(0, 2000) : null, t, t, row.id)
661+ .run();
662+ }
663+
664+ // ---------------------------------------------------------------- workflow: shared
665+
666+ /** Mark an import running. Returns what to do; stops when it was cancelled meanwhile. */
667+ async begin(id: string): Promise<{ kind: "repo" | "bulk"; stop: boolean }> {
668+ const row = await this.mustRow(id);
669+ if (row.status !== "queued" && row.status !== "running") return { kind: row.kind, stop: true };
670+ const owner = await this.dir.account(row.target_owner_id);
671+ const actor = await this.dir.account(row.created_by);
672+ if (!owner || !actor || !(await this.canImportInto(actor, owner))) {
673+ await this.close(row, "failed", "the person or agent who started this import can no longer import into this account");
674+ return { kind: row.kind, stop: true };
675+ }
676+ await this.db.prepare("UPDATE imports SET status = 'running', started_at = COALESCE(started_at, ?), updated_at = ? WHERE id = ?").bind(now(), now(), id).run();
677+ return { kind: row.kind, stop: false };
678+ }
679+
680+ private async context(id: string): Promise<Ctx> {
681+ const row = await this.mustRow(id);
682+ if (row.status !== "running") throw fatal(`import is ${row.status}`);
683+ const token = row.token_ciphertext ? await decryptSecret(this.env.MASTER_KEY, `import:${id}`, row.token_ciphertext) : null;
684+ // Links are keyed by the full normalised API base, so github.com and an Enterprise host never mix.
685+ const apiBase = normalizeApiUrl(row.api_url, this.dev);
686+ const links = await this.db
687+ .prepare("SELECT l.external_id, l.external_login, l.account_id, a.login FROM account_links l JOIN accounts a ON a.id = l.account_id WHERE l.provider = 'github' AND l.api_base = ? AND a.status = 'active'")
688+ .bind(apiBase)
689+ .all<{ external_id: string; external_login: string; account_id: string; login: string }>();
690+ const people = new People(links.results.map((l): Link => ({ externalId: l.external_id, externalLogin: l.external_login, accountId: l.account_id, login: l.login })));
691+ const repo = row.repo_id ? await this.dir.repoById(row.repo_id) : null;
692+ return {
693+ row,
694+ req: parse<StoredRequest>(row.request, {} as StoredRequest),
695+ gh: new GitHubClient(row.api_url, token),
696+ token,
697+ people,
698+ summary: parse<ImportSummary>(row.summary, {}),
699+ steps: parse<Record<string, StepState>>(row.steps, {}),
700+ repo,
701+ stub: repo ? this.env.REPO.get(this.env.REPO.idFromName(repo.id)) : null,
702+ };
703+ }
704+
705+ /** The workflow gave up: record why. */
706+ async fail(id: string, message: string): Promise<void> {
707+ const row = await this.row(id);
708+ if (!row) return;
709+ const steps = parse<Record<string, StepState>>(row.steps, {});
710+ for (const [name, s] of Object.entries(steps)) if (s.status === "running") await this.setStep(id, name, { status: "failed", message: message.slice(0, 300), finishedAt: now() });
711+ await this.close(row, "failed", message);
712+ await this.stopContainer(row);
713+ if (!row.parent_id) await this.notifyDone(row, false);
714+ await this.audit(row, "import.fail", { error: message.slice(0, 500) });
715+ }
716+
717+ private async notifyDone(row: ImportRow, ok: boolean): Promise<void> {
718+ const owner = await this.dir.account(row.target_owner_id);
719+ const target = row.kind === "repo" ? `${owner?.login}/${row.target_name}` : owner?.login;
720+ await this.dir.notify([row.created_by], { repoId: row.repo_id ?? undefined, kind: "import", title: ok ? `Imported ${row.source} into ${target}` : `Import of ${row.source} into ${target} failed`, url: `${this.env.PUBLIC_URL}/imports/${row.id}` });
721+ }
722+
723+ // ---------------------------------------------------------------- workflow: one repository
724+
725+ /** Create the Diffinery repository (or reuse the one this import created on an earlier attempt). */
726+ async stepRepository(id: string): Promise<void> {
727+ const ctx = await this.context(id);
728+ await this.setStep(id, "repository", { status: "running", startedAt: now() });
729+ let gr: GhRepo;
730+ try {
731+ gr = await ctx.gh.get<GhRepo>(`repos/${ctx.row.source}`);
732+ } catch (e) {
733+ if (e instanceof GitHubError && e.status < 500) throw fatal(e.status === 401 ? "GitHub rejected the token (expired or revoked?)" : `GitHub repository ${ctx.row.source} was not found or the token cannot read it`);
734+ throw e;
735+ }
736+ const cloneUrl = gr.clone_url ?? `https://${webHostOf(ctx.row.api_url)}/${gr.full_name}.git`;
737+ if (new URL(cloneUrl).host !== webHostOf(ctx.row.api_url)) throw fatal("GitHub returned a clone URL on another host");
738+ ctx.summary.source = {
739+ htmlUrl: gr.html_url ?? `https://${webHostOf(ctx.row.api_url)}/${gr.full_name}`,
740+ cloneUrl,
741+ defaultBranch: gr.default_branch || "main",
742+ visibility: gr.visibility ?? (gr.private ? "private" : "public"),
743+ archived: !!gr.archived,
744+ description: gr.description ?? "",
745+ homepage: gr.homepage ?? "",
746+ topics: (gr.topics ?? []).slice(0, 20),
747+ };
748+ let repo = ctx.repo;
749+ if (!repo) {
750+ const owner = (await this.dir.account(ctx.row.target_owner_id))!;
751+ const creator = (await this.dir.account(ctx.row.created_by))!;
752+ if (await this.dir.repo(owner.login, ctx.row.target_name!)) throw fatal(`${owner.login}/${ctx.row.target_name} already exists`);
753+ repo = await this.dir.createRepo(owner, creator, { name: ctx.row.target_name!, description: (gr.description ?? "").slice(0, 1000), visibility: visibilityFor(ctx.req.visibility, gr), defaultBranch: ctx.summary.source.defaultBranch }, { authorized: true });
754+ await this.db.prepare("UPDATE imports SET repo_id = ?, updated_at = ? WHERE id = ?").bind(repo.id, now(), id).run();
755+ }
756+ // H2: claim the repository for this import. Refuses if another import holds it, or if the repo
757+ // already holds content this import did not write (so a retry never clobbers an adopted repo).
758+ const stub = this.env.REPO.get(this.env.REPO.idFromName(repo.id));
759+ try {
760+ await stub.importBegin(id);
761+ } catch (e) {
762+ if (e instanceof Error && /409:/.test(e.message)) throw fatal(errMsg(e));
763+ throw e;
764+ }
765+ await this.dir.updateRepo(repo, { homepage: ctx.summary.source.homepage.slice(0, 500) });
766+ await this.dir.setTopics(repo.id, ctx.summary.source.topics);
767+ await this.saveSummary(id, ctx.summary);
768+ await this.setStep(id, "repository", { status: "done", message: `${repo.owner_login}/${repo.name}`, finishedAt: now() });
769+ }
770+
771+ /** Start (or restart) the Git transfer. Returns whether an external agent has to claim it. */
772+ async startGit(id: string): Promise<{ external: boolean; tries: number }> {
773+ const ctx = await this.context(id);
774+ if (!ctx.repo) throw fatal("the repository was not created");
775+ const prev = parse<GitState>(ctx.row.git, {});
776+ const tries = (prev.tries ?? 0) + 1;
777+ const external = this.env.IMPORT_GIT_AGENT === "external";
778+ const git: GitState = { tries, phase: external ? "waiting" : "starting", message: external ? "Waiting for an import agent to claim the transfer" : "Starting the import container", at: now() };
779+ await this.db.prepare("UPDATE imports SET git = ?, updated_at = ? WHERE id = ?").bind(JSON.stringify(git), now(), id).run();
780+ await this.setStep(id, "git", { status: "running", startedAt: ctx.steps.git?.startedAt ?? now(), message: git.message });
781+ if (external) return { external, tries };
782+ if (!this.env.IMPORTER) throw fatal("no import container is configured (IMPORTER binding)");
783+ const token = await this.mintPushToken(ctx.row);
784+ await this.env.IMPORTER.get(this.env.IMPORTER.idFromName(`${id}-${ctx.row.attempt}-${tries}`)).startImport({ importId: id, token, apiUrl: this.env.PUBLIC_URL, stamp: `${ctx.row.attempt}-${tries}` });
785+ return { external, tries };
786+ }
787+
788+ /** A short-lived token for the transfer: the starter's account and scopes, this repository only, one day. */
789+ private async mintPushToken(row: ImportRow): Promise<string> {
790+ if (row.push_token_id) await this.dir.revokeTokenById(row.push_token_id);
791+ const creator = await this.dir.account(row.created_by);
792+ if (!creator) throw fatal("the account that started this import no longer exists");
793+ const req = parse<StoredRequest>(row.request, {} as StoredRequest);
794+ const t = await this.dir.createToken(creator, { name: `GitHub import of ${row.source}`, scopes: req.scopes?.length ? req.scopes : ["write"], repoIds: [row.repo_id!], expiresInDays: 1, importId: row.id });
795+ await this.db.prepare("UPDATE imports SET push_token_id = ?, updated_at = ? WHERE id = ?").bind(t.id, now(), row.id).run();
796+ return t.token;
797+ }
798+
799+ /** Local development and tests: an agent started by hand claims the transfer and gets the push token. */
800+ async claim(id: string, actor: Account): Promise<{ token: string }> {
801+ if (this.env.IMPORT_GIT_AGENT !== "external") throw new HttpError(404, "not found");
802+ const row = await this.row(id);
803+ if (!row || row.created_by !== actor.id) throw new HttpError(404, "import not found");
804+ const git = parse<GitState>(row.git, {});
805+ if (row.status !== "running" || git.phase !== "waiting") throw new HttpError(409, "this import is not waiting for an agent");
806+ const token = await this.mintPushToken(row);
807+ await this.db.prepare("UPDATE imports SET git = json_set(git, '$.phase', 'starting', '$.message', 'Agent claimed the transfer'), updated_at = ? WHERE id = ?").bind(now(), id).run();
808+ return { token };
809+ }
810+
811+ /** Where the transfer stands: done, running, or failed (with whether trying again can help). */
812+ async gitStatus(id: string): Promise<{ state: "running" | "done" | "failed"; error?: string; retryable?: boolean }> {
813+ const row = await this.mustRow(id);
814+ if (row.status !== "running") throw fatal(`import is ${row.status}`);
815+ const git = parse<GitState>(row.git, {});
816+ const r = git.result;
817+ if (!r) return { state: "running" };
818+ if (!r.ok) {
819+ if (!r.secretFindings.length) return { state: "failed", error: r.error ?? "the Git transfer failed", retryable: true };
820+ const kinds = [...new Set(r.secretFindings.map((f) => f.kind))].join(", ");
821+ const stop = parse<StoredRequest>(row.request, {} as StoredRequest).options?.secrets === "stop";
822+ // Retry cannot change an import's options, so the only way to bypass is a fresh import.
823+ const how = stop
824+ ? "The import was set to stop on secrets. Rotate them, then start a NEW import with the secrets option set to bypass (as a person who administers the target); a retry cannot change the options."
825+ : "Only a person who administers the target can import them with an audited bypass (agents and write-scoped tokens cannot); start a new import as such a person.";
826+ return { state: "failed", error: `Push protection found possible secrets in the history (${kinds}). ${how}`, retryable: false };
827+ }
828+ const summary = parse<ImportSummary>(row.summary, {});
829+ const { ok: _o, error: _e, secretFindings, warnings, ...counts } = r;
830+ summary.git = counts;
831+ summary.findings = mergeFindings(summary.findings ?? [], secretFindings);
832+ await this.saveSummary(id, summary);
833+ for (const w of warnings.slice(0, 50)) await this.warn(id, w);
834+ const lfs = r.lfsObjects ? `, ${r.lfsObjects} LFS objects` : "";
835+ await this.setStep(id, "git", { status: "done", count: r.branches + r.tags, message: `${r.branches} branches, ${r.tags} tags, ${r.pullRefs} pull request heads${lfs}`, finishedAt: now() });
836+ return { state: "done" };
837+ }
838+
839+ /** Clear a failed transfer's result before trying again. */
840+ async resetGit(id: string, error: string): Promise<void> {
841+ await this.warn(id, `The Git transfer failed and is being retried: ${error.slice(0, 300)}`);
842+ await this.db.prepare("UPDATE imports SET git = json_remove(git, '$.result'), updated_at = ? WHERE id = ?").bind(now(), id).run();
843+ }
844+
845+ /** One call of a paged step. Rate limits come back as `wait` so the workflow can sleep. */
846+ async runStep(id: string, name: PagedStep, cursor: StepCursor): Promise<{ done: boolean; cursor: StepCursor; wait?: number }> {
847+ const ctx = await this.context(id);
848+ if (!ctx.stub) throw fatal("the repository was not created");
849+ const state = ctx.steps[name];
850+ if (cursor === null || !state || state.status === "pending") await this.setStep(id, name, { status: "running", startedAt: now(), count: 0 });
851+ let out: StepOutcome;
852+ try {
853+ out = await this.step(ctx, name, cursor);
854+ } catch (e) {
855+ if (e instanceof RateLimited) {
856+ await this.setStep(id, name, { message: e.message });
857+ return { done: false, cursor, wait: e.waitSeconds };
858+ }
859+ if (e instanceof GitHubError && e.status < 500 && e.status !== 408) throw fatal(e.message);
860+ throw e;
861+ }
862+ const count = (cursor === null ? 0 : (state?.count ?? 0)) + (out.added ?? 0);
863+ ctx.summary.people = {
864+ linked: [...new Set([...(ctx.summary.people?.linked ?? []), ...ctx.people.mapped])].slice(0, 500),
865+ imported: [...new Set([...(ctx.summary.people?.imported ?? []), ...ctx.people.unmapped])].slice(0, 500),
866+ };
867+ await this.saveSummary(id, ctx.summary);
868+ if (out.done) await this.setStep(id, name, { status: out.skipped ? "skipped" : "done", count, message: out.message ?? `${count}`, finishedAt: now() });
869+ else await this.setStep(id, name, { status: "running", count, message: out.message ?? `${count} so far` });
870+ return { done: out.done, cursor: out.cursor ?? null };
871+ }
872+
873+ private step(ctx: Ctx, name: PagedStep, cursor: StepCursor): Promise<StepOutcome> {
874+ switch (name) {
875+ case "labels":
876+ return this.labels(ctx);
877+ case "issues":
878+ return this.issues(ctx, cursor);
879+ case "comments":
880+ return this.listStep(ctx, cursor, listPath(`repos/${ctx.row.source}/issues/comments`, { sort: "created", direction: "asc" }), (items: GhComment[]) =>
881+ ctx.stub!.importComments(items.map((c) => mapIssueComment(c, ctx.people)).filter((c) => !!c), ctx.row.id),
882+ );
883+ case "reviews":
884+ return this.reviews(ctx, cursor);
885+ case "review_comments":
886+ return this.listStep(ctx, cursor, listPath(`repos/${ctx.row.source}/pulls/comments`, { sort: "created", direction: "asc" }), (items: GhReviewComment[]) =>
887+ ctx.stub!.importComments(items.map((c) => mapReviewComment(c, ctx.people)).filter((c) => !!c), ctx.row.id),
888+ );
889+ case "protection":
890+ return this.protection(ctx);
891+ case "variables":
892+ return this.variables(ctx);
893+ case "releases":
894+ return this.importReleases(ctx, cursor);
895+ }
896+ }
897+
898+ private async labels(ctx: Ctx): Promise<StepOutcome> {
899+ if (!ctx.req.options.labels) return { done: true, skipped: true, message: "Not selected" };
900+ const src = ctx.row.source;
901+ const labels = await ctx.gh.all<GhLabel>(listPath(`repos/${src}/labels`));
902+ await ctx.stub!.importLabels(labels.map((l) => ({ name: l.name, color: l.color ?? "888888", description: l.description ?? "" })), ctx.row.id);
903+ let milestones: GhMilestone[] = [];
904+ try {
905+ milestones = await ctx.gh.all<GhMilestone>(listPath(`repos/${src}/milestones`, { state: "all" }));
906+ } catch (e) {
907+ if (!(e instanceof GitHubError && (e.status === 404 || e.status === 410))) throw e;
908+ }
909+ await ctx.stub!.importMilestones(milestones.map((m) => mapMilestone(m, ctx.people)), ctx.row.id);
910+ return { done: true, added: labels.length + milestones.length, message: `${labels.length} labels, ${milestones.length} milestones` };
911+ }
912+
913+ /** Pull requests first (from /pulls, which has their branches), then issues (from /issues, skipping pull requests). */
914+ private async issues(ctx: Ctx, cursor: StepCursor): Promise<StepOutcome> {
915+ if (!ctx.req.options.issues) return { done: true, skipped: true, message: "Not selected" };
916+ const src = ctx.row.source;
917+ let c = (cursor as { phase: string; next: string } | null) ?? { phase: "pulls", next: listPath(`repos/${src}/pulls`, { state: "all", sort: "created", direction: "asc" }) };
918+ let added = 0;
919+ for (let i = 0; i < PAGES_PER_STEP; i++) {
920+ let page: { items: Array<GhIssue | GhPull>; next: string | null };
921+ try {
922+ page = await ctx.gh.page<GhIssue | GhPull>(c.next);
923+ } catch (e) {
924+ // Issues turned off on GitHub: /issues answers 410.
925+ if (c.phase === "issues" && e instanceof GitHubError && e.status === 410) return { done: true, added, message: undefined };
926+ throw e;
927+ }
928+ const rows = c.phase === "pulls" ? (page.items as GhPull[]).map((p) => mapPull(p, src, ctx.people)) : (page.items as GhIssue[]).filter((x) => !x.pull_request).map((x) => mapIssue(x, ctx.people));
929+ if (rows.length) added += await ctx.stub!.importIssues(rows, ctx.row.id);
930+ if (page.next) c = { phase: c.phase, next: page.next };
931+ else if (c.phase === "pulls") c = { phase: "issues", next: listPath(`repos/${src}/issues`, { state: "all", sort: "created", direction: "asc" }) };
932+ else return { done: true, added };
933+ }
934+ return { done: false, cursor: c, added };
935+ }
936+
937+ /** A repository-wide list endpoint, a few pages per call. */
938+ private async listStep<T>(ctx: Ctx, cursor: StepCursor, first: string, store: (items: T[]) => Promise<number>): Promise<StepOutcome> {
939+ if (!ctx.req.options.issues) return { done: true, skipped: true, message: "Not selected" };
940+ let next = (cursor as string | null) ?? first;
941+ let added = 0;
942+ for (let i = 0; i < PAGES_PER_STEP; i++) {
943+ let page: { items: T[]; next: string | null };
944+ try {
945+ page = await ctx.gh.page<T>(next);
946+ } catch (e) {
947+ // Issues/comments can answer 410 (issues disabled) or 404; that is not a failure, just no data.
948+ if (e instanceof GitHubError && (e.status === 410 || e.status === 404)) return { done: true, added };
949+ throw e;
950+ }
951+ if (page.items.length) added += await store(page.items);
952+ if (!page.next) return { done: true, added };
953+ next = page.next;
954+ }
955+ return { done: false, cursor: next, added };
956+ }
957+
958+ /** Reviews are listed per pull request; merged pull requests also need who merged them (only on the single pull). */
959+ private async reviews(ctx: Ctx, cursor: StepCursor): Promise<StepOutcome> {
960+ if (!ctx.req.options.issues) return { done: true, skipped: true, message: "Not selected" };
961+ const src = ctx.row.source;
962+ const pulls = await ctx.stub!.importPullNumbers(typeof cursor === "number" ? cursor : 0, PULLS_PER_STEP);
963+ let added = 0;
964+ for (const p of pulls) {
965+ let reviews: GhReview[] = [];
966+ try {
967+ reviews = await ctx.gh.all<GhReview>(listPath(`repos/${src}/pulls/${p.number}/reviews`));
968+ } catch (e) {
969+ // A single pull request's reviews may be unavailable (404/410); skip it, don't fail the import.
970+ if (!(e instanceof GitHubError && (e.status === 404 || e.status === 410))) throw e;
971+ await this.warn(ctx.row.id, `Reviews for #${p.number} could not be read and were skipped.`);
972+ }
973+ added += await ctx.stub!.importReviews(reviews.map((r) => mapReview(r, p.number, ctx.people)).filter((r) => !!r), ctx.row.id);
974+ if (p.merged) {
975+ const full = await ctx.gh.get<{ merged_by?: GhUser | null }>(`repos/${src}/pulls/${p.number}`);
976+ if (full.merged_by) await ctx.stub!.importSetMergedBy(p.number, ctx.people.login(full.merged_by));
977+ }
978+ }
979+ if (pulls.length < PULLS_PER_STEP) return { done: true, added };
980+ return { done: false, cursor: pulls[pulls.length - 1]!.number, added };
981+ }
982+
983+ /** Classic branch protection, mapped to Diffinery's rules. Required checks are renamed to the checks Diffinery's workflows report. */
984+ private async protection(ctx: Ctx): Promise<StepOutcome> {
985+ if (!ctx.req.options.protection) return { done: true, skipped: true, message: "Not selected" };
986+ const src = ctx.row.source;
987+ const branches = await ctx.gh.all<{ name: string }>(listPath(`repos/${src}/branches`, { protected: "true" }), 500);
988+ const checkNames = branches.length ? await ctx.stub!.importCheckNames() : {};
989+ const done: string[] = [];
990+ for (const b of branches) {
991+ let p: GhProtection;
992+ try {
993+ p = await ctx.gh.get<GhProtection>(`repos/${src}/branches/${encodeURIComponent(b.name)}/protection`);
994+ } catch (e) {
995+ if (e instanceof GitHubError && (e.status === 403 || e.status === 404)) {
996+ await this.warn(ctx.row.id, `Branch protection of ${b.name} could not be read (the GitHub token needs admin access to the repository); set it up under Settings → Branches.`);
997+ continue;
998+ }
999+ throw e;
1000+ }
1001+ const rules = mapProtection(p);
1002+ rules.requiredChecks = [...new Set(rules.requiredChecks.flatMap((c) => checkNames[c] ?? [c]))];
1003+ await ctx.stub!.importProtection(b.name, rules, ctx.row.id);
1004+ done.push(b.name);
1005+ }
1006+ ctx.summary.protections = done;
1007+ return { done: true, added: done.length, message: done.length ? done.join(", ") : "No protected branches" };
1008+ }
1009+
1010+ /**
1011+ * Whether this import may write Actions variables now. Agents can never manage variables, and a
1012+ * human's authority over the target is re-checked at the moment the step runs (not only at create
1013+ * time), so a person who lost org admin in the meantime cannot push variables through an import.
1014+ */
1015+ private async mayManageVariables(ctx: Ctx): Promise<{ ok: true } | { ok: false; reason: string }> {
1016+ const creator = await this.dir.account(ctx.row.created_by);
1017+ if (!creator || creator.kind === "agent") return { ok: false, reason: "skipped (agents cannot manage variables)" };
1018+ const owner = await this.dir.account(ctx.row.target_owner_id);
1019+ if (!owner || !(await this.canImportInto(creator, owner))) return { ok: false, reason: "skipped (you no longer administer the target)" };
1020+ return { ok: true };
1021+ }
1022+
1023+ /** Actions variables (values) and secrets (names only: GitHub never returns secret values). */
1024+ private async variables(ctx: Ctx): Promise<StepOutcome> {
1025+ if (!ctx.req.options.variables) return { done: true, skipped: true, message: "Not selected" };
1026+ const may = await this.mayManageVariables(ctx);
1027+ if (!may.ok) return { done: true, skipped: true, message: may.reason };
1028+ const src = ctx.row.source;
1029+ const read = async <T>(path: string, what: string): Promise<T[]> => {
1030+ try {
1031+ return await ctx.gh.all<T>(path, 1000);
1032+ } catch (e) {
1033+ if (e instanceof GitHubError && (e.status === 403 || e.status === 404)) {
1034+ await this.warn(ctx.row.id, `Actions ${what} could not be read (the GitHub token needs admin access, or "Variables"/"Secrets" read permission for fine-grained tokens).`);
1035+ return [];
1036+ }
1037+ throw e;
1038+ }
1039+ };
1040+ const vars = (await read<{ name: string; value: string }>(`repos/${src}/actions/variables?per_page=30`, "variables")).filter((v) => validVarName(v.name));
1041+ for (const v of vars) await ctx.stub!.setVariable(v.name, v.value);
1042+ const secrets = await read<{ name: string }>(`repos/${src}/actions/secrets?per_page=100`, "secrets");
1043+ ctx.summary.variables = vars.map((v) => v.name);
1044+ ctx.summary.secretNames = secrets.map((s) => s.name);
1045+ const env = await this.importEnvironments(ctx, read);
1046+ const envMsg = env ? `, ${env.imported} environments` : "";
1047+ return { done: true, added: vars.length, message: `${vars.length} variables, ${secrets.length} secret names${envMsg}` };
1048+ }
1049+
1050+ /**
1051+ * Deployment environments: names, their branch policy, and their variables.
1052+ * Environment secret *names* are listed in the summary but their values are
1053+ * never copied (an admin re-enters them), the same policy as repository
1054+ * secrets. Skipped (with a note) where RepoDO has no environments feature yet.
1055+ */
1056+ private async importEnvironments(ctx: Ctx, read: <T>(path: string, what: string) => Promise<T[]>): Promise<{ imported: number; variables: number; secretNames: string[] } | null> {
1057+ const stub = ctx.stub as unknown as EnvImporter | null;
1058+ if (!stub) return null;
1059+ const caller = this.importCaller(ctx);
1060+ const src = ctx.row.source;
1061+ const envs = await read<GhEnvironment>(`repos/${src}/environments?per_page=100`, "environments");
1062+ if (!envs.length) return null;
1063+ let variables = 0;
1064+ const secretNames: string[] = [];
1065+ try {
1066+ for (const e of envs) {
1067+ if (!e.name || typeof e.name !== "string" || e.name.includes("\0")) continue;
1068+ const policy = e.deployment_branch_policy ?? null;
1069+ await stub.upsertEnvironment(e.name, { deployment_branch_policy: policy }, caller);
1070+ if (policy?.custom_branch_policies) {
1071+ const patterns = await read<{ name?: string; type?: string }>(`repos/${src}/environments/${encodeURIComponent(e.name)}/deployment-branch-policies`, "branch policies");
1072+ for (const p of patterns) if (p.name) await stub.addEnvironmentBranchPattern(e.name, { name: p.name, type: p.type }, caller);
1073+ }
1074+ const envVars = (await read<{ name: string; value: string }>(`repos/${src}/environments/${encodeURIComponent(e.name)}/variables?per_page=100`, "environment variables")).filter((v) => validVarName(v.name));
1075+ for (const v of envVars) {
1076+ await stub.setEnvironmentVariable(e.name, v.name, v.value, caller);
1077+ variables++;
1078+ }
1079+ const envSecrets = await read<{ name: string }>(`repos/${src}/environments/${encodeURIComponent(e.name)}/secrets?per_page=100`, "environment secrets");
1080+ for (const s of envSecrets) secretNames.push(`${e.name}/${s.name}`);
1081+ }
1082+ } catch (e) {
1083+ // RepoDO without an environments feature (the separate branch has not landed): skip quietly.
1084+ if (methodMissing(e)) return null;
1085+ throw e;
1086+ }
1087+ const summary = { imported: envs.length, variables, secretNames };
1088+ ctx.summary.environments = summary;
1089+ return summary;
1090+ }
1091+
1092+ /**
1093+ * Releases and their assets. Release rows and asset bytes live in RepoDO's
1094+ * Releases module (`importRelease`/`importReleaseAsset`), which is delivered
1095+ * by a separate branch. Until that lands this step detects the methods are
1096+ * absent and records that the tags were imported with the Git data instead.
1097+ *
1098+ * Each asset is handed to RepoDO as a `{ url }` source on a GitHub host, with
1099+ * the import token in an Authorization header; RepoDO streams the body with a
1100+ * byte cap and drops the credential on any redirect to another host, so the
1101+ * asset bytes are never buffered in this Worker.
1102+ */
1103+ private async importReleases(ctx: Ctx, cursor: StepCursor): Promise<StepOutcome> {
1104+ if (!ctx.req.options.releases) return { done: true, skipped: true, message: "Not selected" };
1105+ const stub = ctx.stub as unknown as ReleaseImporter | null;
1106+ if (!stub) throw fatal("the repository was not created");
1107+ const caller = this.importCaller(ctx);
1108+ // The release asset host must share the API base's origin before the token is attached.
1109+ const apiOrigin = new URL(ctx.row.api_url).origin;
1110+ let next = (cursor as string | null) ?? listPath(`repos/${ctx.row.source}/releases`);
1111+ const totals = ctx.summary.releases ?? { imported: 0, assets: 0, skippedAssets: 0 };
1112+ try {
1113+ for (let i = 0; i < PAGES_PER_STEP; i++) {
1114+ const page = await ctx.gh.page<GhRelease>(next);
1115+ for (const r of page.items) {
1116+ if (!r.tag_name) continue;
1117+ const { id: releaseId } = await stub.importRelease(
1118+ { tag_name: r.tag_name, target_commitish: r.target_commitish, name: r.name ?? r.tag_name, body: r.body ?? "", draft: !!r.draft, prerelease: !!r.prerelease, created_at: r.created_at, published_at: r.published_at, author: this.importedActor(ctx, r.author), latest: false },
1119+ caller,
1120+ );
1121+ totals.imported++;
1122+ for (const a of r.assets ?? []) {
1123+ const name = safeAssetName(a.name);
1124+ if (!name) {
1125+ totals.skippedAssets++;
1126+ await this.warn(ctx.row.id, `Release ${r.tag_name}: skipped an asset with an unsafe name "${a.name}".`);
1127+ continue;
1128+ }
1129+ const headers: Record<string, string> = { accept: "application/octet-stream", "user-agent": "Diffinery-Importer" };
1130+ // Low 3: only attach the token when the asset URL is on the same origin as the API base.
1131+ // On a redirect to another host, RepoDO drops the credential; we never add it cross-origin here.
1132+ let sameOrigin = false;
1133+ try {
1134+ sameOrigin = new URL(a.url).origin === apiOrigin;
1135+ } catch {
1136+ sameOrigin = false;
1137+ }
1138+ if (ctx.token && sameOrigin) headers.authorization = `Bearer ${ctx.token}`;
1139+ try {
1140+ await stub.importReleaseAsset(releaseId, { name, label: a.label ?? undefined, content_type: a.content_type, download_count: a.download_count, created_at: a.created_at, updated_at: a.updated_at, uploader: this.importedActor(ctx, a.uploader) }, { url: a.url, headers }, caller);
1141+ totals.assets++;
1142+ } catch (e) {
1143+ if (methodMissing(e)) throw e;
1144+ totals.skippedAssets++;
1145+ await this.warn(ctx.row.id, `Release ${r.tag_name}: could not import asset "${name}": ${errMsg(e)}`);
1146+ }
1147+ }
1148+ }
1149+ ctx.summary.releases = totals;
1150+ if (!page.next) {
1151+ if (totals.imported === 0) return { done: true, skipped: true, message: "No releases" };
1152+ return { done: true, added: totals.imported, message: `${totals.imported} releases, ${totals.assets} assets${totals.skippedAssets ? `, ${totals.skippedAssets} skipped` : ""}` };
1153+ }
1154+ next = page.next;
1155+ }
1156+ } catch (e) {
1157+ // RepoDO without a releases feature (the separate branch has not landed): the tags came
1158+ // with the Git data, so record that and move on rather than failing the import.
1159+ if (methodMissing(e) && totals.imported === 0) {
1160+ await this.warn(ctx.row.id, "Releases were not imported: this deployment has no releases feature yet. Their tags were imported with the Git data.");
1161+ return { done: true, skipped: true, message: "Not supported here (tags were imported)" };
1162+ }
1163+ throw e;
1164+ }
1165+ return { done: false, cursor: next, added: 0, message: `${totals.imported} releases so far` };
1166+ }
1167+
1168+ /** A writer caller for the import's own writes to the new repository (the person or agent who started it). */
1169+ private importCaller(ctx: Ctx): ImportCaller {
1170+ return { id: ctx.row.created_by, login: ctx.repo?.owner_login ?? "importer", kind: "user", role: "admin", importId: ctx.row.id };
1171+ }
1172+
1173+ /** Attribute an imported release/asset to a linked account, or to the GitHub user if unlinked. */
1174+ private importedActor(ctx: Ctx, u: GhUser | null | undefined): { id?: string; login: string; kind?: string } {
1175+ const p = ctx.people.person(u);
1176+ return { id: p.id, login: p.login, kind: p.kind };
1177+ }
1178+
1179+ /**
1180+ * Last step: pull request refs, cleanup, default branch, archived state, and the summary.
1181+ * The GitHub token is deleted and the push token revoked.
1182+ */
1183+ async finalize(id: string): Promise<void> {
1184+ const ctx = await this.context(id);
1185+ if (!ctx.repo || !ctx.stub) throw fatal("the repository was not created");
1186+ await this.setStep(id, "finalize", { status: "running", startedAt: now() });
1187+ // The server creates refs/pull/N/head from the imported pull requests; then the import's own refs can go.
1188+ const pullRefs = await ctx.stub.importPullRefs(id);
1189+ ctx.summary.pullRefs = { created: pullRefs.created, missing: pullRefs.missing.slice(0, 200) };
1190+ if (pullRefs.missing.length) await this.warn(id, `${pullRefs.missing.length} pull requests have no head commit (their fork was deleted on GitHub): ${pullRefs.missing.slice(0, 20).map((n) => `#${n}`).join(", ")}`);
1191+ await ctx.stub.importDropTempRefs(id);
1192+ const git = parse<GitState>(ctx.row.git, {});
1193+ const patch: { defaultBranch?: string; archived?: boolean } = {};
1194+ if (git.result?.defaultBranch && git.result.defaultBranch !== ctx.repo.default_branch) patch.defaultBranch = git.result.defaultBranch;
1195+ if (ctx.summary.source?.archived) patch.archived = true;
1196+ if (Object.keys(patch).length) await this.dir.updateRepo(ctx.repo, patch);
1197+ const counts = await ctx.stub.importCounts();
1198+ ctx.summary.counts = { issues: counts.issues, pulls: counts.pulls, comments: counts.comments, reviews: counts.reviews, labels: counts.labels, milestones: counts.milestones };
1199+ ctx.summary.findings = mergeFindings(ctx.summary.findings ?? [], (await ctx.stub.importFindings()).map((f) => ({ kind: f.kind, blob: f.blob })));
1200+ await this.saveSummary(id, ctx.summary);
1201+ // Release the import lock: the repository is now open for normal writes.
1202+ await ctx.stub.importEnd(id);
1203+ await this.setStep(id, "finalize", { status: "done", finishedAt: now(), message: `${counts.issues} issues, ${counts.pulls} pull requests` });
1204+ await this.close(ctx.row, "succeeded", null);
1205+ await this.audit(ctx.row, "import.finish", { repo: `${ctx.repo.owner_login}/${ctx.repo.name}`, counts: ctx.summary.counts, findings: ctx.summary.findings.length });
1206+ if (!ctx.row.parent_id) await this.notifyDone(ctx.row, true);
1207+ }
1208+
1209+ // ---------------------------------------------------------------- workflow: bulk imports
1210+
1211+ /** Organization variables and secret names, when importing an organization into an organization. */
1212+ async bulkOrganization(id: string): Promise<{ wait?: number }> {
1213+ const ctx = await this.context(id);
1214+ await this.setStep(id, "organization", { status: "running", startedAt: now() });
1215+ const owner = await this.dir.account(ctx.row.target_owner_id);
1216+ if (!ctx.req.options.variables || ctx.req.sourceType !== "Organization" || owner?.kind !== "org") {
1217+ await this.setStep(id, "organization", { status: "skipped", message: owner?.kind !== "org" ? "Importing into a personal account" : "Not selected", finishedAt: now() });
1218+ return {};
1219+ }
1220+ // H1: agents can never manage variables; a human's org admin is re-checked at run time.
1221+ const may = await this.mayManageVariables(ctx);
1222+ if (!may.ok) {
1223+ await this.setStep(id, "organization", { status: "skipped", message: may.reason, finishedAt: now() });
1224+ return {};
1225+ }
1226+ const read = async <T>(path: string, what: string): Promise<T[]> => {
1227+ try {
1228+ return await ctx.gh.all<T>(path, 1000);
1229+ } catch (e) {
1230+ if (e instanceof RateLimited) throw e;
1231+ if (e instanceof GitHubError && (e.status === 403 || e.status === 404)) {
1232+ await this.warn(id, `Organization ${what} could not be read (the GitHub token needs organization admin access).`);
1233+ return [];
1234+ }
1235+ throw e;
1236+ }
1237+ };
1238+ try {
1239+ const vars = (await read<{ name: string; value: string; visibility?: string }>(`orgs/${ctx.row.source}/actions/variables?per_page=30`, "variables")).filter((v) => validVarName(v.name));
1240+ const secrets = await read<{ name: string }>(`orgs/${ctx.row.source}/actions/secrets?per_page=100`, "secrets");
1241+ // Add only variables the organization does not already have; never overwrite an admin's value.
1242+ const { added, skipped } = vars.length ? await this.dir.addOrgVariablesIfAbsent(owner.id, Object.fromEntries(vars.map((v) => [v.name, v.value]))) : { added: [], skipped: [] };
1243+ for (const name of added) await this.audit(ctx.row, "org.variable_set", { org: owner.login, name });
1244+ for (const name of skipped) await this.warn(id, `Organization variable ${name} already exists and was left unchanged.`);
1245+ ctx.summary.orgVariables = added;
1246+ ctx.summary.orgSecretNames = secrets.map((s) => s.name);
1247+ await this.saveSummary(id, ctx.summary);
1248+ const skip = skipped.length ? `, ${skipped.length} kept` : "";
1249+ await this.setStep(id, "organization", { status: "done", count: added.length, message: `${added.length} variables added${skip}, ${secrets.length} secret names`, finishedAt: now() });
1250+ return {};
1251+ } catch (e) {
1252+ if (e instanceof RateLimited) return { wait: e.waitSeconds };
1253+ throw e;
1254+ }
1255+ }
1256+
1257+ /** Start queued repository imports of a bulk import, up to its concurrency. */
1258+ async startChildren(parentId: string): Promise<number> {
1259+ const parent = await this.mustRow(parentId);
1260+ if (parent.status !== "running") return 0;
1261+ const req = parse<StoredRequest>(parent.request, {} as StoredRequest);
1262+ const running = (await this.db.prepare("SELECT count(*) AS n FROM imports WHERE parent_id = ? AND status = 'running'").bind(parentId).first<{ n: number }>())?.n ?? 0;
1263+ let started = 0;
1264+ for (let i = running; i < (req.concurrency || 3); i++) {
1265+ // Claim one queued child atomically, so parallel finishers never start the same one twice.
1266+ const next = await this.db
1267+ .prepare("UPDATE imports SET status = 'running', started_at = ?, updated_at = ? WHERE id = (SELECT id FROM imports WHERE parent_id = ? AND status = 'queued' ORDER BY created_at, source LIMIT 1) AND status = 'queued' RETURNING id, attempt")
1268+ .bind(now(), now(), parentId)
1269+ .first<{ id: string; attempt: number }>();
1270+ if (!next) break;
1271+ try {
1272+ await this.launch(next.id, next.attempt);
1273+ started++;
1274+ } catch (e) {
1275+ await this.fail(next.id, `could not start: ${errMsg(e)}`);
1276+ }
1277+ }
1278+ await this.bulkProgress(parentId);
1279+ return started;
1280+ }
1281+
1282+ /** Counts of a bulk import's repositories, written to its "repositories" step. */
1283+ async bulkProgress(parentId: string): Promise<{ open: number }> {
1284+ const rows = (await this.db.prepare("SELECT status, count(*) AS n FROM imports WHERE parent_id = ? GROUP BY status").bind(parentId).all<{ status: ImportStatus; n: number }>()).results;
1285+ const n = (s: ImportStatus) => rows.find((r) => r.status === s)?.n ?? 0;
1286+ const total = rows.reduce((a, r) => a + r.n, 0);
1287+ const open = n("queued") + n("running");
1288+ const finished = total - open;
1289+ await this.setStep(parentId, "repositories", { status: open ? "running" : "done", count: finished, message: `${finished} of ${total} finished (${n("succeeded")} imported, ${n("failed")} failed)` });
1290+ return { open };
1291+ }
1292+
1293+ /** After a repository import of a bulk import ends: start the next one, or wake the parent when none are left. */
1294+ async afterChild(id: string): Promise<void> {
1295+ const row = await this.row(id);
1296+ if (!row?.parent_id) return;
1297+ const parent = await this.row(row.parent_id);
1298+ if (!parent || parent.status !== "running") return;
1299+ await this.startChildren(parent.id);
1300+ const { open } = await this.bulkProgress(parent.id);
1301+ if (!open) {
1302+ try {
1303+ await (await this.env.IMPORT_WORKFLOW.get(`${parent.id}-${parent.attempt}`)).sendEvent({ type: "children-done", payload: {} });
1304+ } catch (e) {
1305+ console.error("waking bulk import failed", parent.id, e);
1306+ }
1307+ }
1308+ }
1309+
1310+ async bulkFinish(id: string): Promise<void> {
1311+ const row = await this.mustRow(id);
1312+ const children = await this.children(id);
1313+ const count = (s: ImportStatus) => children.filter((c) => c.status === s).length;
1314+ const summary = parse<ImportSummary>(row.summary, {});
1315+ summary.children = { total: children.length, succeeded: count("succeeded"), failed: count("failed"), cancelled: count("cancelled"), open: count("queued") + count("running") };
1316+ await this.saveSummary(id, summary);
1317+ await this.setStep(id, "finalize", { status: "done", finishedAt: now(), message: `${summary.children.succeeded} of ${summary.children.total} repositories imported` });
1318+ // Children still open at finalize (a transfer that never finished within the cap) count as not imported,
1319+ // so a bulk import never reports success while a repository import is still running.
1320+ const notImported = summary.children.failed + summary.children.cancelled + summary.children.open;
1321+ await this.close(row, notImported ? "failed" : "succeeded", notImported ? `${notImported} of ${children.length} repositories were not imported` : null);
1322+ await this.audit(row, "import.finish", { children: summary.children });
1323+ await this.notifyDone(row, !notImported);
1324+ }
1325+
1326+ // ---------------------------------------------------------------- the Git transfer agent
1327+
1328+ /** The agent's credentials: this import's own *current* push token, while the import runs. */
1329+ private async agentRow(id: string, importId: string | undefined, tokenId: string | undefined): Promise<ImportRow> {
1330+ if (importId !== id) throw new HttpError(403, "this token does not belong to this import");
1331+ const row = await this.row(id);
1332+ if (!row || row.status !== "running" || !row.repo_id) throw new HttpError(409, "this import is not running");
1333+ // M4: only the import's current push token (this attempt's) is accepted; a stale token is refused.
1334+ if (!tokenId || row.push_token_id !== tokenId) throw new HttpError(403, "this push token is no longer current for the import");
1335+ return row;
1336+ }
1337+
1338+ async agentSpec(id: string, importId: string | undefined, tokenId: string | undefined, pushToken: string): Promise<ImportSpec> {
1339+ const row = await this.agentRow(id, importId, tokenId);
1340+ const summary = parse<ImportSummary>(row.summary, {});
1341+ const req = parse<StoredRequest>(row.request, {} as StoredRequest);
1342+ const repo = await this.dir.repoById(row.repo_id!);
1343+ if (!repo || !summary.source) throw new HttpError(409, "the repository was not created");
1344+ const cloneUrl = new URL(summary.source.cloneUrl);
1345+ if (cloneUrl.host !== webHostOf(row.api_url)) throw new HttpError(409, "the clone URL is not on the GitHub host being imported from");
1346+ await this.db.prepare("UPDATE imports SET git = json_set(COALESCE(git, '{}'), '$.phase', 'clone', '$.message', ?, '$.at', ?), updated_at = ? WHERE id = ?").bind(`Cloning ${row.source}`, now(), now(), id).run();
1347+ return {
1348+ importId: id,
1349+ source: { fullName: row.source, cloneUrl: cloneUrl.toString(), defaultBranch: summary.source.defaultBranch },
1350+ githubToken: row.token_ciphertext ? await decryptSecret(this.env.MASTER_KEY, `import:${id}`, row.token_ciphertext) : null,
1351+ target: { fullName: `${repo.owner_login}/${repo.name}`, pushUrl: `${this.env.PUBLIC_URL}/${repo.owner_login}/${repo.name}.git` },
1352+ pushToken,
1353+ lfs: req.options.lfs,
1354+ secrets: req.options.secrets,
1355+ bypassReason: `Imported from GitHub ${row.source}`,
1356+ chunkBytes: DEFAULT_CHUNK_BYTES,
1357+ };
1358+ }
1359+
1360+ async agentProgress(id: string, importId: string | undefined, tokenId: string | undefined, p: ImportProgress): Promise<void> {
1361+ await this.agentRow(id, importId, tokenId);
1362+ const n = (v: unknown) => (typeof v === "number" && Number.isFinite(v) && v >= 0 ? Math.floor(v) : 0);
1363+ await this.db
1364+ .prepare("UPDATE imports SET git = json_set(COALESCE(git, '{}'), '$.phase', ?, '$.message', ?, '$.chunks', ?, '$.bytes', ?, '$.at', ?), updated_at = ? WHERE id = ?")
1365+ .bind(String(p.phase ?? "").slice(0, 20), String(p.message ?? "").slice(0, 300), n(p.chunks), n(p.bytes), now(), now(), id)
1366+ .run();
1367+ }
1368+
1369+ /** The agent's result. The push token is revoked now, and the workflow is woken. */
1370+ async agentComplete(id: string, importId: string | undefined, tokenId: string | undefined, raw: ImportGitResult): Promise<void> {
1371+ const row = await this.agentRow(id, importId, tokenId);
1372+ const n = (v: unknown) => (typeof v === "number" && Number.isFinite(v) && v >= 0 ? Math.floor(v) : 0);
1373+ const s = (v: unknown, max: number) => (typeof v === "string" ? v.slice(0, max) : undefined);
1374+ const result: ImportGitResult = {
1375+ ok: raw?.ok === true,
1376+ error: s(raw?.error, 2000),
1377+ branches: n(raw?.branches),
1378+ tags: n(raw?.tags),
1379+ pullRefs: n(raw?.pullRefs),
1380+ chunks: n(raw?.chunks),
1381+ bytes: n(raw?.bytes),
1382+ lfsObjects: n(raw?.lfsObjects),
1383+ lfsBytes: n(raw?.lfsBytes),
1384+ defaultBranch: s(raw?.defaultBranch, 255) ?? null,
1385+ secretFindings: (Array.isArray(raw?.secretFindings) ? raw.secretFindings : []).slice(0, 100).map((f) => ({ kind: s(f?.kind, 100) ?? "secret", blob: s(f?.blob, 64) ?? "", path: s(f?.path, 500) })),
1386+ warnings: (Array.isArray(raw?.warnings) ? raw.warnings : []).slice(0, 50).map((w) => s(w, 500) ?? ""),
1387+ };
1388+ await this.db
1389+ .prepare("UPDATE imports SET git = json_set(COALESCE(git, '{}'), '$.result', json(?), '$.phase', ?, '$.message', ?, '$.at', ?), updated_at = ? WHERE id = ?")
1390+ .bind(JSON.stringify(result), result.ok ? "done" : "failed", result.ok ? "Git data copied" : (result.error ?? "failed").slice(0, 300), now(), now(), id)
1391+ .run();
1392+ if (row.push_token_id) await this.dir.revokeTokenById(row.push_token_id);
1393+ await this.wake(row);
1394+ }
1395+
1396+ /** The import container stopped. If its agent never reported, the transfer failed (and may be retried). */
1397+ async gitStopped(id: string, exitCode: number | null, stamp?: string): Promise<void> {
1398+ const row = await this.row(id);
1399+ if (!row || row.status !== "running") return;
1400+ const git = parse<GitState>(row.git, {});
1401+ // Ignore a late stop from an earlier attempt or try (M4 / Correctness 7): only the current one may fail the transfer.
1402+ if (stamp && stamp !== `${row.attempt}-${git.tries ?? 0}`) return;
1403+ if (git.result) return;
1404+ const result = { ok: false, error: `the import container stopped before finishing (exit code ${exitCode ?? "unknown"})`, branches: 0, tags: 0, pullRefs: 0, chunks: 0, bytes: 0, lfsObjects: 0, lfsBytes: 0, defaultBranch: null, secretFindings: [], warnings: [] } satisfies ImportGitResult;
1405+ await this.db.prepare("UPDATE imports SET git = json_set(COALESCE(git, '{}'), '$.result', json(?)), updated_at = ? WHERE id = ?").bind(JSON.stringify(result), now(), id).run();
1406+ if (row.push_token_id) await this.dir.revokeTokenById(row.push_token_id);
1407+ await this.wake(row);
1408+ }
1409+
1410+ private async wake(row: ImportRow): Promise<void> {
1411+ try {
1412+ await (await this.env.IMPORT_WORKFLOW.get(`${row.id}-${row.attempt}`)).sendEvent({ type: "git-done", payload: {} });
1413+ } catch (e) {
1414+ // The workflow also checks on its own every few minutes.
1415+ console.error("waking import failed", row.id, e);
1416+ }
1417+ }
1418+
1419+ // ---------------------------------------------------------------- linked GitHub accounts
1420+
1421+ async links(accountId: string): Promise<Array<{ id: string; host: string; apiBase: string; login: string; externalId: string; createdAt: string }>> {
1422+ const r = await this.db.prepare("SELECT * FROM account_links WHERE account_id = ? ORDER BY created_at").bind(accountId).all<Record<string, string>>();
1423+ return r.results.map((l) => ({ id: l.id!, host: webHostOf(l.api_base!), apiBase: l.api_base!, login: l.external_login!, externalId: l.external_id!, createdAt: l.created_at! }));
1424+ }
1425+
1426+ /**
1427+ * A per-account, per-host nonce the person puts in a PUBLIC gist to prove they control the GitHub
1428+ * account. Derived (HMAC of the master key) so it needs no storage and is stable until linked.
1429+ */
1430+ async linkChallenge(account: Account, apiUrlRaw: string | null): Promise<{ nonce: string; apiBase: string; host: string; gistsUrl: string }> {
1431+ if (account.kind !== "user") throw new HttpError(403, "only people can link GitHub accounts");
1432+ const apiBase = normalizeApiUrl(apiUrlRaw, this.dev);
1433+ const nonce = await this.linkNonce(account.id, apiBase);
1434+ return { nonce, apiBase, host: webHostOf(apiBase), gistsUrl: `${apiBase}/gists` };
1435+ }
1436+
1437+ private async linkNonce(accountId: string, apiBase: string): Promise<string> {
1438+ const key = await crypto.subtle.importKey("raw", new TextEncoder().encode(this.env.MASTER_KEY ?? "dev"), { name: "HMAC", hash: "SHA-256" }, false, ["sign"]);
1439+ const mac = await crypto.subtle.sign("HMAC", key, new TextEncoder().encode(`github-link:${accountId}:${apiBase}`));
1440+ return `diffinery-link-${[...new Uint8Array(mac)].slice(0, 16).map((b) => b.toString(16).padStart(2, "0")).join("")}`;
1441+ }
1442+
1443+ /**
1444+ * Link a GitHub account by gist proof (no token): the person's nonce must appear in one of their
1445+ * PUBLIC gists, read unauthenticated. The numeric id comes from the public user profile, so a
1446+ * later login rename cannot silently re-point the link.
1447+ */
1448+ async linkByGist(account: Account, login: string, apiUrlRaw: string | null): Promise<{ id: string; host: string; login: string }> {
1449+ if (account.kind !== "user") throw new HttpError(403, "only people can link GitHub accounts");
1450+ if (!/^[A-Za-z0-9](?:[A-Za-z0-9-]{0,38})$/.test(login)) throw new HttpError(422, "that is not a valid GitHub login");
1451+ const apiBase = normalizeApiUrl(apiUrlRaw, this.dev);
1452+ const nonce = await this.linkNonce(account.id, apiBase);
1453+ const gh = new GitHubClient(apiBase, null); // unauthenticated: public data only
1454+ let gists: Array<{ id: string; description?: string | null; public?: boolean }>;
1455+ let user: { login: string; id: number };
1456+ try {
1457+ gists = await gh.get<typeof gists>(`users/${login}/gists?per_page=100`);
1458+ user = await gh.get<{ login: string; id: number }>(`users/${login}`);
1459+ } catch (e) {
1460+ if (e instanceof GitHubError && e.status === 404) throw new HttpError(422, `GitHub user ${login} was not found`);
1461+ if (e instanceof RateLimited) throw new HttpError(429, e.message);
1462+ throw new HttpError(502, `could not reach GitHub: ${errMsg(e)}`);
1463+ }
1464+ if (!user?.login || user.id === undefined) throw new HttpError(502, "GitHub did not return a user");
1465+ const proven = await this.gistHasNonce(gh, gists, nonce);
1466+ if (!proven) throw new HttpError(422, `create a PUBLIC gist whose description or a file contains ${nonce}, then try again`);
1467+ return this.saveLink(account, apiBase, String(user.id), user.login);
1468+ }
1469+
1470+ /**
1471+ * Whether one of the user's public gists contains the nonce. The gist bodies are read through the
1472+ * same API host (GET /gists/{id}), never a cross-host raw URL, so no credential or request leaves
1473+ * the allowlisted host.
1474+ */
1475+ private async gistHasNonce(gh: GitHubClient, gists: Array<{ id: string; description?: string | null; public?: boolean }>, nonce: string): Promise<boolean> {
1476+ let fetched = 0;
1477+ for (const g of gists.slice(0, 100)) {
1478+ if (g.public === false || !g.id) continue;
1479+ if ((g.description ?? "").includes(nonce)) return true;
1480+ if (fetched >= 20) continue; // bound how many gist bodies we read
1481+ fetched++;
1482+ try {
1483+ const full = await gh.get<{ files?: Record<string, { content?: string | null }> }>(`gists/${encodeURIComponent(g.id)}`);
1484+ for (const f of Object.values(full.files ?? {})) if ((f.content ?? "").includes(nonce)) return true;
1485+ } catch {
1486+ // skip an unreadable gist
1487+ }
1488+ }
1489+ return false;
1490+ }
1491+
1492+ private async saveLink(account: Account, apiBase: string, externalId: string, login: string): Promise<{ id: string; host: string; login: string }> {
1493+ const existing = await this.db.prepare("SELECT * FROM account_links WHERE provider = 'github' AND api_base = ? AND external_id = ?").bind(apiBase, externalId).first<Record<string, string>>();
1494+ if (existing && existing.account_id !== account.id) throw new HttpError(409, `GitHub user ${login} is linked to another Diffinery account`);
1495+ const id = existing?.id ?? newId();
1496+ await this.db
1497+ .prepare("INSERT INTO account_links (id, account_id, provider, api_base, external_id, external_login, created_at) VALUES (?, ?, 'github', ?, ?, ?, ?) ON CONFLICT(provider, api_base, external_id) DO UPDATE SET external_login = excluded.external_login")
1498+ .bind(id, account.id, apiBase, externalId, login, now())
1499+ .run();
1500+ await this.dir.audit({ actorId: account.id, actorLogin: account.login, action: "account.link_github", target: `${webHostOf(apiBase)}/${login}` });
1501+ return { id, host: webHostOf(apiBase), login };
1502+ }
1503+
1504+ // ---------------------------------------------------------------- optional OAuth web flow
1505+
1506+ /** Whether a GitHub OAuth app is configured for the web link flow. */
1507+ get oauthEnabled(): boolean {
1508+ return !!(this.env.GITHUB_OAUTH_CLIENT_ID && this.env.GITHUB_OAUTH_CLIENT_SECRET);
1509+ }
1510+
1511+ /** The GitHub authorize URL with a state bound (HMAC) to this account, so a callback cannot be replayed for another. */
1512+ async oauthStart(account: Account): Promise<string> {
1513+ if (!this.oauthEnabled) throw new HttpError(404, "GitHub OAuth is not configured");
1514+ const state = await this.oauthState(account.id);
1515+ const u = new URL("https://github.com/login/oauth/authorize");
1516+ u.searchParams.set("client_id", this.env.GITHUB_OAUTH_CLIENT_ID!);
1517+ u.searchParams.set("redirect_uri", `${this.env.PUBLIC_URL}/settings/linked/github/callback`);
1518+ u.searchParams.set("scope", "read:user");
1519+ u.searchParams.set("state", state);
1520+ u.searchParams.set("allow_signup", "false");
1521+ return u.toString();
1522+ }
1523+
1524+ private async oauthState(accountId: string): Promise<string> {
1525+ const key = await crypto.subtle.importKey("raw", new TextEncoder().encode(this.env.MASTER_KEY ?? "dev"), { name: "HMAC", hash: "SHA-256" }, false, ["sign"]);
1526+ const mac = await crypto.subtle.sign("HMAC", key, new TextEncoder().encode(`github-oauth:${accountId}`));
1527+ return `${accountId}.${[...new Uint8Array(mac)].slice(0, 16).map((b) => b.toString(16).padStart(2, "0")).join("")}`;
1528+ }
1529+
1530+ async oauthCallback(account: Account, code: string, state: string): Promise<{ id: string; host: string; login: string }> {
1531+ if (!this.oauthEnabled) throw new HttpError(404, "GitHub OAuth is not configured");
1532+ if (!code || !state || state !== (await this.oauthState(account.id))) throw new HttpError(422, "invalid or expired OAuth state");
1533+ const tokenRes = await fetch("https://github.com/login/oauth/access_token", {
1534+ method: "POST",
1535+ headers: { accept: "application/json", "content-type": "application/json" },
1536+ body: JSON.stringify({ client_id: this.env.GITHUB_OAUTH_CLIENT_ID, client_secret: this.env.GITHUB_OAUTH_CLIENT_SECRET, code, redirect_uri: `${this.env.PUBLIC_URL}/settings/linked/github/callback` }),
1537+ });
1538+ const tok = (await tokenRes.json().catch(() => ({}))) as { access_token?: string };
1539+ if (!tok.access_token) throw new HttpError(422, "GitHub did not return an access token");
1540+ let user: { login: string; id: number };
1541+ try {
1542+ user = await new GitHubClient("https://api.github.com", tok.access_token).get<{ login: string; id: number }>("user");
1543+ } catch (e) {
1544+ throw new HttpError(502, `could not read the GitHub user: ${errMsg(e)}`);
1545+ }
1546+ if (!user?.login || user.id === undefined) throw new HttpError(502, "GitHub did not return a user");
1547+ return this.saveLink(account, "https://api.github.com", String(user.id), user.login);
1548+ }
1549+
1550+ async unlink(account: Account, id: string): Promise<void> {
1551+ await this.db.prepare("DELETE FROM account_links WHERE id = ? AND account_id = ?").bind(id, account.id).run();
1552+ await this.dir.audit({ actorId: account.id, actorLogin: account.login, action: "account.unlink_github", target: id });
1553+ }
1554+
1555+ /** Site-admin action: clear any account's GitHub link (e.g. after an account change of hands). Audited. */
1556+ async clearLink(admin: Account, id: string): Promise<void> {
1557+ if (!admin.site_admin) throw new HttpError(403, "site admin only");
1558+ const link = await this.db.prepare("SELECT * FROM account_links WHERE id = ?").bind(id).first<Record<string, string>>();
1559+ if (!link) throw new HttpError(404, "link not found");
1560+ await this.db.prepare("DELETE FROM account_links WHERE id = ?").bind(id).run();
1561+ await this.dir.audit({ actorId: admin.id, actorLogin: admin.login, action: "account.link_cleared", target: `${webHostOf(link.api_base!)}/${link.external_login}`, detail: { account: link.account_id } });
1562+ }
1563+}
1564+
1565+/** Findings from the agent (with paths) and from the server's push protection (full blob ids), one per blob. */
1566+function mergeFindings(a: ImportSecretFinding[], b: ImportSecretFinding[]): ImportSecretFinding[] {
1567+ const out = [...a];
1568+ for (const f of b) {
1569+ const same = out.find((x) => x.kind === f.kind && (x.blob.startsWith(f.blob) || f.blob.startsWith(x.blob)));
1570+ if (!same) out.push(f);
1571+ else {
1572+ if (f.blob.length > same.blob.length) same.blob = f.blob;
1573+ same.path ??= f.path;
1574+ }
1575+ }
1576+ return out.slice(0, 200);
1577+}
1578+
1579+interface GhRepo {
1580+ name: string;
1581+ full_name: string;
1582+ owner?: { login: string };
1583+ private?: boolean;
1584+ visibility?: string;
1585+ fork?: boolean;
1586+ archived?: boolean;
1587+ description?: string | null;
1588+ homepage?: string | null;
1589+ topics?: string[];
1590+ default_branch?: string;
1591+ clone_url?: string;
1592+ html_url?: string;
1593+ /** Approximate size of the repository on GitHub, in KB. */
1594+ size?: number;
1595+}
addedapps/diffinery/src/imports/workflow.ts+97−0View file
@@ -1,0 +1,97 @@
1+import { WorkflowEntrypoint, type WorkflowEvent, type WorkflowStep } from "cloudflare:workers";
2+import type { Env } from "../env.js";
3+import { ImportService, PAGED_STEPS, type StepCursor } from "./service.js";
4+
5+export interface ImportParams {
6+ importId: string;
7+}
8+
9+const RETRY = { retries: { limit: 5, delay: "15 seconds", backoff: "exponential" }, timeout: "15 minutes" } as const;
10+/** How long the workflow waits between checks on the Git transfer, and for how many checks (6 hours). */
11+const GIT_CHECK = "10 minutes";
12+const GIT_CHECKS = 36;
13+const GIT_TRIES = 2;
14+/** An overall cap on how many times one paged step may be called, so a hostile or broken source cannot run the workflow forever. */
15+const MAX_STEP_CALLS = 2000;
16+
17+/**
18+ * Durable execution of one GitHub import (instance id `<importId>-<attempt>`).
19+ * A repository import runs its steps in order, each retried on failure and
20+ * paged so that no step runs long; rate limits become sleeps. A bulk import
21+ * fans out one repository import per repository, a few at a time: each one
22+ * starts the next when it ends, and the last one wakes the bulk import.
23+ */
24+export class ImportWorkflow extends WorkflowEntrypoint<Env, ImportParams> {
25+ override async run(event: WorkflowEvent<ImportParams>, step: WorkflowStep): Promise<unknown> {
26+ const id = event.payload.importId;
27+ const svc = () => new ImportService(this.env);
28+ const begin = await step.do("begin", RETRY, () => svc().begin(id));
29+ if (begin.stop) return { stopped: true };
30+ try {
31+ if (begin.kind === "bulk") await this.bulk(id, step, svc);
32+ else await this.repository(id, step, svc);
33+ } catch (e) {
34+ const message = e instanceof Error ? e.message : String(e);
35+ await step.do("fail", RETRY, () => svc().fail(id, message));
36+ }
37+ await step.do("after", RETRY, () => svc().afterChild(id));
38+ return { done: true };
39+ }
40+
41+ private async repository(id: string, step: WorkflowStep, svc: () => ImportService): Promise<void> {
42+ await step.do("repository", RETRY, () => svc().stepRepository(id));
43+
44+ for (let attempt = 1; ; attempt++) {
45+ await step.do(`git start ${attempt}`, RETRY, () => svc().startGit(id));
46+ let status: Awaited<ReturnType<ImportService["gitStatus"]>> = { state: "running" };
47+ for (let i = 0; i < GIT_CHECKS; i++) {
48+ status = await step.do(`git check ${attempt}.${i}`, RETRY, () => svc().gitStatus(id));
49+ if (status.state !== "running") break;
50+ try {
51+ await step.waitForEvent(`git wait ${attempt}.${i}`, { type: "git-done", timeout: GIT_CHECK });
52+ } catch {
53+ // timed out: check again
54+ }
55+ }
56+ if (status.state === "done") break;
57+ if (status.state === "running") throw new Error("the Git transfer did not finish within 6 hours");
58+ if (!status.retryable || attempt >= GIT_TRIES) throw new Error(status.error ?? "the Git transfer failed");
59+ const error = status.error ?? "";
60+ await step.do(`git reset ${attempt}`, RETRY, () => svc().resetGit(id, error));
61+ }
62+
63+ for (const name of PAGED_STEPS) {
64+ let cursor: StepCursor = null;
65+ for (let i = 0; ; i++) {
66+ if (i >= MAX_STEP_CALLS) throw new Error(`the ${name} step did not finish within ${MAX_STEP_CALLS} pages`);
67+ const r = await step.do(`${name} ${i}`, RETRY, () => svc().runStep(id, name, cursor));
68+ if (r.wait) await step.sleep(`${name} rate limit ${i}`, `${r.wait} seconds`);
69+ if (r.done) break;
70+ cursor = r.cursor;
71+ }
72+ }
73+
74+ await step.do("finalize", RETRY, () => svc().finalize(id));
75+ }
76+
77+ private async bulk(id: string, step: WorkflowStep, svc: () => ImportService): Promise<void> {
78+ for (let i = 0; ; i++) {
79+ const r = await step.do(`organization ${i}`, RETRY, () => svc().bulkOrganization(id));
80+ if (!r.wait) break;
81+ await step.sleep(`organization rate limit ${i}`, `${r.wait} seconds`);
82+ }
83+ await step.do("start repositories", RETRY, () => svc().startChildren(id));
84+ // Repository imports start each other; this only waits for the last one (and checks hourly, up to a week).
85+ for (let i = 0; i < 168; i++) {
86+ const { open } = await step.do(`progress ${i}`, RETRY, () => svc().bulkProgress(id));
87+ if (!open) break;
88+ try {
89+ await step.waitForEvent(`wait ${i}`, { type: "children-done", timeout: "1 hour" });
90+ } catch {
91+ // timed out: restart anything stuck in the queue, then check again
92+ await step.do(`restart ${i}`, RETRY, () => svc().startChildren(id));
93+ }
94+ }
95+ await step.do("finalize", RETRY, () => svc().bulkFinish(id));
96+ }
97+}
modifiedapps/diffinery/src/index.ts+2−0View file
@@ -8,6 +8,8 @@
88 export { RepoDO } from "./repo/RepoDO.js";
99 export { RunnerContainer, SandboxContainer } from "./runner/containers.js";
1010 export { RunWorkflow } from "./runner/workflow.js";
11+export { ImportWorkflow } from "./imports/workflow.js";
12+export { ImportContainer } from "./imports/container.js";
1113
1214 // The web UI is mounted last so API, git and runner routes take precedence;
1315 // unknown API paths answer in JSON rather than falling through to its HTML 404.
modifiedapps/diffinery/src/lib/directory.ts+96−11View file
@@ -39,6 +39,8 @@
3939 pushed_at: string | null;
4040 /** JSON array of topic names (GitHub's repository topics). */
4141 topics?: string;
42+ /** Project website (GitHub's `homepage`). */
43+ homepage?: string;
4244 }
4345
4446 export type TokenScope = "read" | "write" | "admin";
@@ -296,21 +298,24 @@
296298
297299 // ---------------------------------------------------------------- tokens
298300
299- async createToken(account: Account, input: { name: string; scopes?: TokenScope[]; repoIds?: string[] | null; expiresInDays?: number }): Promise<{ token: string; id: string; expiresAt: string }> {
301+ /** `importId`: a GitHub import's short-lived push token (see imports/service.ts). */
302+ async createToken(account: Account, input: { name: string; scopes?: TokenScope[]; repoIds?: string[] | null; expiresInDays?: number; importId?: string }): Promise<{ token: string; id: string; expiresAt: string }> {
300303 const days = Math.min(Math.max(input.expiresInDays ?? 90, 1), 366);
301304 const token = `dfy_${randomToken(30)}`;
302305 const id = newId();
303306 const expiresAt = new Date(Date.now() + days * 86400_000).toISOString();
304307 await this.db
305- .prepare("INSERT INTO tokens (id, account_id, name, token_hash, prefix, scopes, repo_ids, expires_at, created_at) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)")
306- .bind(id, account.id, input.name || "token", await sha256(token), token.slice(0, 10), JSON.stringify(input.scopes?.length ? input.scopes : ["write"]), input.repoIds ? JSON.stringify(input.repoIds) : null, expiresAt, now())
308+ .prepare("INSERT INTO tokens (id, account_id, name, token_hash, prefix, scopes, repo_ids, expires_at, created_at, import_id) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)")
309+ .bind(id, account.id, input.name || "token", await sha256(token), token.slice(0, 10), JSON.stringify(input.scopes?.length ? input.scopes : ["write"]), input.repoIds ? JSON.stringify(input.repoIds) : null, expiresAt, now(), input.importId ?? null)
307310 .run();
308- await tokenCreated(this.env, this, account, { name: input.name || "token", scopes: input.scopes?.length ? input.scopes : ["write"] }).catch((e) => console.error("token email failed", e));
311+ // An import's own push token is server-managed and short-lived: no "new token" email.
312+ if (!input.importId) await tokenCreated(this.env, this, account, { name: input.name || "token", scopes: input.scopes?.length ? input.scopes : ["write"] }).catch((e) => console.error("token email failed", e));
309313 return { token, id, expiresAt };
310314 }
311315
312316 async tokens(accountId: string): Promise<Array<{ id: string; name: string; prefix: string; scopes: string[]; repoIds: string[] | null; expiresAt: string; lastUsedAt: string | null; createdAt: string }>> {
313- const r = await this.db.prepare("SELECT * FROM tokens WHERE account_id = ? AND revoked_at IS NULL ORDER BY created_at DESC").bind(accountId).all<Record<string, string | null>>();
317+ // Import push tokens (import_id set) are server-managed and hidden from the personal token list.
318+ const r = await this.db.prepare("SELECT * FROM tokens WHERE account_id = ? AND revoked_at IS NULL AND import_id IS NULL ORDER BY created_at DESC").bind(accountId).all<Record<string, string | null>>();
314319 return r.results.map((t) => ({ id: t.id!, name: t.name!, prefix: t.prefix!, scopes: JSON.parse(t.scopes!), repoIds: t.repo_ids ? JSON.parse(t.repo_ids) : null, expiresAt: t.expires_at!, lastUsedAt: t.last_used_at ?? null, createdAt: t.created_at! }));
315320 }
316321
@@ -318,7 +323,7 @@
318323 await this.db.prepare("UPDATE tokens SET revoked_at = ? WHERE id = ? AND account_id = ?").bind(now(), id, accountId).run();
319324 }
320325
321- async verifyToken(token: string): Promise<{ account: Account; scopes: TokenScope[]; repoIds: string[] | null } | null> {
326+ async verifyToken(token: string): Promise<{ id: string; account: Account; scopes: TokenScope[]; repoIds: string[] | null; importId: string | null } | null> {
322327 const row = await this.db.prepare("SELECT * FROM tokens WHERE token_hash = ?").bind(await sha256(token)).first<Record<string, string | null>>();
323328 if (!row || row.revoked_at || row.expires_at! < now()) return null;
324329 const account = await this.account(row.account_id!);
@@ -326,9 +331,19 @@
326331 if (!row.last_used_at || Date.now() - Date.parse(row.last_used_at) > 3600_000) {
327332 await this.db.prepare("UPDATE tokens SET last_used_at = ? WHERE id = ?").bind(now(), row.id!).run();
328333 }
329- return { account, scopes: JSON.parse(row.scopes!), repoIds: row.repo_ids ? JSON.parse(row.repo_ids) : null };
334+ return { id: row.id!, account, scopes: JSON.parse(row.scopes!), repoIds: row.repo_ids ? JSON.parse(row.repo_ids) : null, importId: row.import_id ?? null };
330335 }
331336
337+ /** Revoke a token by id whoever owns it (server-managed tokens such as an import's push token). */
338+ async revokeTokenById(id: string): Promise<void> {
339+ await this.db.prepare("UPDATE tokens SET revoked_at = ? WHERE id = ? AND revoked_at IS NULL").bind(now(), id).run();
340+ }
341+
342+ /** Revoke every live token an import ever minted (not only the current one), so a stale token can never push. */
343+ async revokeTokensByImport(importId: string): Promise<void> {
344+ await this.db.prepare("UPDATE tokens SET revoked_at = ? WHERE import_id = ? AND revoked_at IS NULL").bind(now(), importId).run();
345+ }
346+
332347 // ---------------------------------------------------------------- invites
333348
334349 async createInvite(inviter: Account, input: { email: string; orgId?: string; repoId?: string; role: string }): Promise<{ id: string; url: string; token: string }> {
@@ -463,11 +478,14 @@
463478 ).results;
464479 }
465480
466- async createRepo(owner: Account, creator: Account, input: { name: string; description?: string; visibility?: string; defaultBranch?: string }): Promise<Repo> {
481+ /** `o.authorized`: the caller already checked that `creator` may create repositories for `owner` (GitHub imports). */
482+ async createRepo(owner: Account, creator: Account, input: { name: string; description?: string; visibility?: string; defaultBranch?: string }, o: { authorized?: boolean } = {}): Promise<Repo> {
467483 if (!isValidName(input.name)) throw new HttpError(422, "invalid repository name: use letters, numbers, '.', '-' and '_'");
468484 // Like GitHub, a member who creates an organization repository administers it, whatever the base permission.
469485 let creatorGrant: string | null = null;
470- if (owner.kind === "org") {
486+ if (o.authorized) {
487+ // checked by the caller
488+ } else if (owner.kind === "org") {
471489 const subject = creator.kind === "agent" ? creator.owner_id! : creator.id;
472490 const role = await this.orgRole(owner.id, subject);
473491 if (!role && !creator.site_admin) throw new HttpError(403, "you are not a member of this organization");
@@ -484,16 +502,27 @@
484502 ...(creatorGrant ? [this.db.prepare("INSERT INTO repo_access (repo_id, account_id, role) VALUES (?, ?, 'admin')").bind(id, creatorGrant)] : []),
485503 ]);
486504 const meta: RepoMeta = { repoId: id, ownerId: owner.id, owner: owner.login, name: input.name, defaultBranch: branch, ownerKind: owner.kind === "org" ? "org" : "user" };
487- await this.env.REPO.get(this.env.REPO.idFromName(id)).init(meta);
505+ const stub = this.env.REPO.get(this.env.REPO.idFromName(id));
506+ await stub.init(meta);
507+ if (owner.kind === "org") {
508+ const vars = await this.orgVariables(owner.id);
509+ if (vars.length) await stub.setOrgVariables(Object.fromEntries(vars.map((v) => [v.name, v.value])));
510+ }
488511 return (await this.repoById(id))!;
489512 }
490513
491- async updateRepo(repo: Repo, patch: { name?: string; description?: string; visibility?: string; defaultBranch?: string; archived?: boolean }): Promise<Repo> {
514+ async updateRepo(repo: Repo, patch: { name?: string; description?: string; visibility?: string; defaultBranch?: string; archived?: boolean; homepage?: string; topics?: string[] }): Promise<Repo> {
492515 if (patch.name && !isValidName(patch.name)) throw new HttpError(422, "invalid repository name");
493516 await this.db
494517 .prepare("UPDATE repos SET name = COALESCE(?, name), description = COALESCE(?, description), visibility = COALESCE(?, visibility), default_branch = COALESCE(?, default_branch), archived = COALESCE(?, archived), updated_at = ? WHERE id = ?")
495518 .bind(patch.name ?? null, patch.description ?? null, repoVisibility(patch.visibility) ?? null, patch.defaultBranch ?? null, patch.archived === undefined ? null : patch.archived ? 1 : 0, now(), repo.id)
496519 .run();
520+ if (patch.homepage !== undefined || patch.topics !== undefined) {
521+ await this.db
522+ .prepare("UPDATE repos SET homepage = COALESCE(?, homepage), topics = COALESCE(?, topics) WHERE id = ?")
523+ .bind(patch.homepage ?? null, patch.topics ? JSON.stringify(patch.topics.slice(0, 20)) : null, repo.id)
524+ .run();
525+ }
497526 if (patch.name || patch.defaultBranch) await this.env.REPO.get(this.env.REPO.idFromName(repo.id)).updateMeta({ name: patch.name ?? repo.name, defaultBranch: patch.defaultBranch ?? repo.default_branch });
498527 return (await this.repoById(repo.id))!;
499528 }
@@ -508,6 +537,62 @@
508537 await this.db.prepare("UPDATE repos SET topics = ?, updated_at = ? WHERE id = ?").bind(JSON.stringify(topics), now(), repoId).run();
509538 }
510539
540+ // ---------------------------------------------------------------- organization variables
541+
542+ async orgVariables(orgId: string): Promise<Array<{ name: string; value: string; updated_at: string }>> {
543+ return (await this.db.prepare("SELECT name, value, updated_at FROM org_variables WHERE org_id = ? ORDER BY name").bind(orgId).all<{ name: string; value: string; updated_at: string }>()).results;
544+ }
545+
546+ /** Set (or, with null, delete) organization variables, then copy the set into each of the organization's repositories. */
547+ async setOrgVariables(orgId: string, changes: Record<string, string | null>): Promise<void> {
548+ const stmts = Object.entries(changes).map(([name, value]) =>
549+ value === null
550+ ? this.db.prepare("DELETE FROM org_variables WHERE org_id = ? AND name = ?").bind(orgId, name.toUpperCase())
551+ : this.db.prepare("INSERT OR REPLACE INTO org_variables (org_id, name, value, updated_at) VALUES (?, ?, ?, ?)").bind(orgId, name.toUpperCase(), value, now()),
552+ );
553+ if (stmts.length) await this.db.batch(stmts);
554+ await this.syncOrgVariables(orgId);
555+ }
556+
557+ /**
558+ * Add organization variables only where none exists (ON CONFLICT DO NOTHING), used by imports so
559+ * an import never overwrites a value an org admin already set. Returns which names were added and
560+ * which were skipped because a variable of that name already existed.
561+ */
562+ async addOrgVariablesIfAbsent(orgId: string, vars: Record<string, string>): Promise<{ added: string[]; skipped: string[] }> {
563+ const before = new Set((await this.orgVariables(orgId)).map((v) => v.name));
564+ const added: string[] = [];
565+ const skipped: string[] = [];
566+ const stmts: D1PreparedStatement[] = [];
567+ for (const [rawName, value] of Object.entries(vars)) {
568+ const name = rawName.toUpperCase();
569+ if (before.has(name)) {
570+ skipped.push(name);
571+ continue;
572+ }
573+ added.push(name);
574+ stmts.push(this.db.prepare("INSERT INTO org_variables (org_id, name, value, updated_at) VALUES (?, ?, ?, ?) ON CONFLICT(org_id, name) DO NOTHING").bind(orgId, name, value, now()));
575+ }
576+ if (stmts.length) {
577+ await this.db.batch(stmts);
578+ await this.syncOrgVariables(orgId);
579+ }
580+ return { added, skipped };
581+ }
582+
583+ /** Copy the organization's current variable set into each of its repositories' Durable Objects. */
584+ private async syncOrgVariables(orgId: string): Promise<void> {
585+ const vars = Object.fromEntries((await this.orgVariables(orgId)).map((v) => [v.name, v.value]));
586+ const repos = await this.db.prepare("SELECT id FROM repos WHERE owner_id = ?").bind(orgId).all<{ id: string }>();
587+ for (const r of repos.results) await this.env.REPO.get(this.env.REPO.idFromName(r.id)).setOrgVariables(vars);
588+ }
589+
590+ /** The highest role an agent may ever hold (its policy's maxRole, default write). */
591+ agentMaxRole(account: Account): Role {
592+ const policy = account.agent_policy ? (JSON.parse(account.agent_policy) as { maxRole?: Role }) : {};
593+ return policy.maxRole ?? "write";
594+ }
595+
511596 async touchPushed(repoId: string): Promise<void> {
512597 await this.db.prepare("UPDATE repos SET pushed_at = ?, updated_at = ? WHERE id = ?").bind(now(), now(), repoId).run();
513598 }
modifiedapps/diffinery/src/lib/util.ts+1−1View file
@@ -67,7 +67,7 @@
6767 }
6868
6969 /** Top-level paths the app serves; no account may use them as its login. */
70-export const RESERVED_LOGINS = new Set(["api", "mcp", "runner", "settings", "new", "notifications", "invite", "request-access", "healthz", "twirp", "login", "logout", "admin", "explore", "favicon.ico", "robots.txt", "about", "docs", "static", "assets", "www", "git", "diffinery", "auth", "signin", "signup", "security", "v2", "-"]);
70+export const RESERVED_LOGINS = new Set(["api", "mcp", "runner", "settings", "new", "notifications", "invite", "request-access", "healthz", "twirp", "login", "logout", "admin", "explore", "favicon.ico", "robots.txt", "about", "docs", "static", "assets", "www", "git", "diffinery", "auth", "signin", "signup", "security", "v2", "imports", "-"]);
7171
7272 export const isValidLogin = (s: string): boolean => isValidName(s) && !s.includes(".") && !RESERVED_LOGINS.has(s.toLowerCase());
7373
modifiedapps/diffinery/src/mcp/server.ts+60−0View file
@@ -3,9 +3,12 @@
33 import { requireAccount, requireScope, type AppContext, type AppVars } from "../auth/principal.js";
44 import { HttpError, b64, fromB64 } from "../lib/util.js";
55 import type { Role } from "../repo/types.js";
6+import type { TokenScope } from "../lib/directory.js";
67 import { loadRepo } from "../http/common.js";
78 import { RELEASE_TOOLS } from "./releases.js";
89 import { DOCS_TOOLS } from "./docs.js";
10+import { ImportService } from "../imports/service.js";
11+import { importJson } from "../http/imports.js";
912
1013 /**
1114 * Model Context Protocol server (Streamable HTTP transport, JSON responses).
@@ -373,8 +376,65 @@
373376 },
374377 ...RELEASE_TOOLS,
375378 ...DOCS_TOOLS,
379+ start_github_import: {
380+ description:
381+ "Start a server-side import of a GitHub repository into Diffinery: git history (keeping issue and pull-request numbers), issues and pull requests with comments and reviews, labels, milestones, branch protection, Actions variables, releases and deployment environments. Secret VALUES are never read from GitHub (only their names are reported, to set by hand), and push protection is never bypassed on an agent's behalf. You can import only into your owner's account or an organization you or they administer. Returns the import id; poll it with get_import.",
382+ input: {
383+ source: { type: "string", description: "GitHub repository as owner/name or a github.com/Enterprise URL" },
384+ owner: { type: "string", description: "Diffinery user or organization to import into" },
385+ name: { type: "string", description: "Target repository name (default: the GitHub name)" },
386+ visibility: { type: "string", enum: ["keep", "private", "public"], description: "Visibility of the new repository (default: keep GitHub's; GitHub internal repositories become private)" },
387+ github_token: { type: "string", description: "A GitHub token to read a private repository (optional)" },
388+ github_api_url: { type: "string", description: "GitHub Enterprise REST API base, e.g. https://ghe.example.com/api/v3 (optional)" },
389+ },
390+ required: ["source", "owner"],
391+ run: async (c, a) => {
392+ const me = requireAccount(c);
393+ const svc = new ImportService(c.env);
394+ const row = await svc.create(me, importScopes(c), {
395+ source: str(a, "source"),
396+ owner: str(a, "owner"),
397+ name: optStr(a, "name"),
398+ visibility: optStr(a, "visibility"),
399+ github_token: optStr(a, "github_token"),
400+ github_api_url: optStr(a, "github_api_url"),
401+ });
402+ return importJson(c.env, svc, row, { children: true });
403+ },
404+ },
405+ get_import: {
406+ description: "Get an import you started: its status, per-step progress, warnings, any error, and (for an organization import) its repositories.",
407+ input: { id: { type: "string", description: "The import id" } },
408+ required: ["id"],
409+ readOnly: true,
410+ run: async (c, a) => {
411+ const me = requireAccount(c);
412+ const svc = new ImportService(c.env);
413+ const row = await svc.row(str(a, "id"));
414+ if (!row || !(await svc.visibleTo(row, me))) throw new HttpError(404, "import not found");
415+ return importJson(c.env, svc, row, { children: true });
416+ },
417+ },
418+ list_imports: {
419+ description: "List the imports you have started, newest first.",
420+ input: {},
421+ readOnly: true,
422+ run: async (c) => {
423+ const me = requireAccount(c);
424+ const svc = new ImportService(c.env);
425+ const rows = await svc.list(me.id);
426+ return Promise.all(rows.map((r) => importJson(c.env, svc, r)));
427+ },
428+ },
376429 };
377430
431+/** The token's scopes, used to authorize starting an import (needs write or admin). */
432+function importScopes(c: AppContext): TokenScope[] {
433+ const p = c.get("principal");
434+ if (p.type === "account" && p.importId) throw new HttpError(403, "this token can only be used by its GitHub import");
435+ return p.type === "account" ? p.scopes : [];
436+}
437+
378438 async function sandboxCall(c: AppContext, a: Args, path: string, payload: unknown) {
379439 const { stub, caller } = await repoOf(c, a, "write");
380440 const id = str(a, "sandbox_id");
modifiedapps/diffinery/src/repo/RepoDO.ts+177−5View file
@@ -45,6 +45,7 @@
4545 import { Deployments } from "./deployments.js";
4646 import { triggerWorkflows } from "./extra-triggers.js";
4747 import { GitData } from "./gitdata.js";
48+import { IMPORT_REF_PREFIX, RepoImports } from "./imports.js";
4849 import { RepoObjectStore } from "./objects.js";
4950 import { MergeQueue, QUEUE_BRANCH_PREFIX, normalizeMergeQueue } from "./queue.js";
5051 import { Releases } from "./releases.js";
@@ -248,6 +249,16 @@
248249 return reportStatus(req, "push exceeds 100 MB; split it into smaller pushes", []);
249250 }
250251 const req = parseReceivePack(body);
252+ // `git push -o diffinery-import` from a GitHub import's own token: no workflows, webhooks,
253+ // pull request syncs or notifications (see apps/diffinery/src/imports).
254+ const importMode = !!caller.importId && req.pushOptions.some((o) => o.trim() === "diffinery-import");
255+ // H2: while an import holds the repository, only that import may push. Everything else waits
256+ // until the import finalises, so protection, guardrails, CI and webhooks are never bypassed.
257+ const lockHolder = this.importLockHolder();
258+ if (lockHolder && (!importMode || caller.importId !== lockHolder)) {
259+ const denied = req.commands.map((c) => ({ ref: c.ref, ok: false as const, reason: "repository is locked by an import in progress" }));
260+ return reportStatus(req, "repository is locked by an import in progress", denied);
261+ }
251262 const messages: string[] = [];
252263 let unpackError: string | null = null;
253264 // Like git's quarantine: nothing from the pack is stored until push protection passes,
@@ -277,7 +288,8 @@
277288 if (reason && caller.role === "admin" && caller.kind === "user") {
278289 messages.push(`Push protection bypassed by ${caller.login}: ${reason}`);
279290 for (const l of leaks.slice(0, 10)) messages.push(` ${l.kind} in blob ${l.oid.slice(0, 10)}`);
280- this.emit({ type: "audit", action: "push.secret_bypassed", actor: caller, target, detail: { kinds, reason, refs: req.commands.map((c) => c.ref) }, at: now() });
291+ this.emit({ type: "audit", action: "push.secret_bypassed", actor: caller, target, detail: { kinds, reason, refs: req.commands.map((c) => c.ref), ...(importMode ? { import: caller.importId, blobs: leaks.slice(0, 50).map((l) => l.oid) } : {}) }, at: now() });
292+ if (importMode) this.importsModule.recordFindings(leaks);
281293 } else {
282294 messages.push("Push blocked: possible secrets detected.");
283295 for (const l of leaks.slice(0, 10)) messages.push(` ${l.kind} in blob ${l.oid.slice(0, 10)}`);
@@ -298,9 +310,10 @@
298310
299311 // Validate every command before applying any, so `atomic` is honoured.
300312 const checks: Array<RefResult & { cmd: (typeof req.commands)[number] }> = [];
301- for (const cmd of req.commands) checks.push({ cmd, ...(await this.checkRefUpdate(cmd.ref, cmd.oldOid, cmd.newOid, caller)) });
313+ for (const cmd of req.commands) checks.push({ cmd, ...(await this.checkRefUpdate(cmd.ref, cmd.oldOid, cmd.newOid, caller, importMode)) });
302314 const atomic = req.capabilities.has("atomic");
303315 const anyFailed = checks.some((c) => !c.ok);
316+ let imported = 0;
304317 for (const c of checks) {
305318 if (!c.ok) {
306319 results.push({ ref: c.ref, ok: false, reason: c.reason });
@@ -312,8 +325,15 @@
312325 }
313326 const applied = this.casRef(c.cmd.ref, c.cmd.oldOid, c.cmd.newOid);
314327 results.push(applied ? { ref: c.ref, ok: true } : { ref: c.ref, ok: false, reason: "fetch first (stale info)" });
315- if (applied) await this.afterRefUpdate(c.cmd.ref, c.cmd.oldOid, c.cmd.newOid, caller, messages);
328+ if (applied && importMode) {
329+ imported++;
330+ // Record provenance so a later push may move this ref only if the import still owns it.
331+ if (c.cmd.newOid === ZERO_OID) this.sql.exec("DELETE FROM import_refs WHERE ref = ?", c.cmd.ref);
332+ else this.importsModule.recordImportRef(c.cmd.ref, c.cmd.newOid, caller.importId!);
333+ } else if (applied) await this.afterRefUpdate(c.cmd.ref, c.cmd.oldOid, c.cmd.newOid, caller, messages);
316334 }
335+ // One audit entry per import push instead of push events.
336+ if (imported) this.emit({ type: "audit", action: "import.push", actor: caller, target: `${this.meta.owner}/${this.meta.name}`, detail: { import: caller.importId, refs: imported }, at: now() });
317337 return reportStatus(req, null, results, messages);
318338 }
319339
@@ -324,14 +344,29 @@
324344 return row ? (JSON.parse(row.rules) as ProtectionRules) : null;
325345 }
326346
327- /** Server-side rules for a ref update: permissions, protection, fast-forward, agent guardrails. */
328- async checkRefUpdate(ref: string, oldOid: string, newOid: string, caller: Caller): Promise<RefResult> {
347+ /**
348+ * Server-side rules for a ref update: permissions, protection, fast-forward, agent guardrails.
349+ * `importMode`: a GitHub import filling the repository it created, which is the only writer
350+ * until it finishes, so branch rules (copied from GitHub at its end) don't apply to it.
351+ */
352+ async checkRefUpdate(ref: string, oldOid: string, newOid: string, caller: Caller, importMode = false): Promise<RefResult> {
329353 if (!isValidRefName(ref)) return { ref, ok: false, reason: "invalid ref name" };
330354 if (!atLeast(caller.role, "write")) return { ref, ok: false, reason: "permission denied" };
331355 const deleting = newOid === ZERO_OID;
332356 if (ref.startsWith("refs/pull/")) return { ref, ok: false, reason: "refs/pull/ is managed by Diffinery for pull requests" };
333357 if (ref.startsWith(`refs/heads/${QUEUE_BRANCH_PREFIX}`)) return { ref, ok: false, reason: "merge queue branches are read-only" };
334358 if (!deleting && !this.store.has(newOid)) return { ref, ok: false, reason: "missing objects" };
359+ if (ref.startsWith(IMPORT_REF_PREFIX) && !importMode) return { ref, ok: false, reason: `${IMPORT_REF_PREFIX} is reserved for GitHub imports` };
360+ if (importMode) {
361+ if (!/^refs\/(heads|tags|diffinery-import)\//.test(ref)) return { ref, ok: false, reason: "imports write branches and tags only" };
362+ // H2: create a ref, or move one this import previously wrote — never a ref adopted since.
363+ const current = this.getRef(ref);
364+ if (current && !deleting) {
365+ const owned = this.importsModule.importRefOid(ref, caller.importId!);
366+ if (owned !== current) return { ref, ok: false, reason: "import may not overwrite a ref it did not write" };
367+ }
368+ return { ref, ok: true };
369+ }
335370 // A workflow's GITHUB_TOKEN never adds or changes workflow files (GitHub's `workflows` permission).
336371 const workflowEdit = await this.policy.refuseWorkflowEdit(caller, oldOid, newOid);
337372 if (workflowEdit) return { ref, ok: false, reason: workflowEdit };
@@ -1244,6 +1279,143 @@
12441279 await triggerWorkflows(this, { name: "repository_dispatch", action: eventType, ref: this.headRef, sha, payload, title: eventType }, caller);
12451280 }
12461281
1282+ // ---------------------------------------------------------------- RPC passthroughs: GitHub imports and milestones (repo/imports.ts)
1283+
1284+ private importsCache?: RepoImports;
1285+ private get importsModule(): RepoImports {
1286+ return (this.importsCache ??= new RepoImports(this.sql));
1287+ }
1288+
1289+ /** The import that currently holds this repository, or null. */
1290+ importLockHolder(): string | null {
1291+ return this.sql.exec<{ value: string }>("SELECT value FROM meta WHERE key = 'import_lock'").toArray()[0]?.value ?? null;
1292+ }
1293+
1294+ /**
1295+ * Claim the repository for an import. Refuses if another import holds it, or if the repo
1296+ * already holds any ref, issue, comment, milestone or protection this import did not write
1297+ * (so a retry never adopts content people added in the meantime — H2).
1298+ */
1299+ importBegin(importId: string): void {
1300+ const holder = this.importLockHolder();
1301+ if (holder && holder !== importId) throw new HttpError(409, "another import is in progress for this repository");
1302+ // A ref not written by this import, or any foreign row, means the repo was adopted since.
1303+ for (const r of this.refs()) {
1304+ if (r.name.startsWith("refs/diffinery-import/")) continue;
1305+ if (this.importsModule.importRefOid(r.name, importId) === null) {
1306+ if (holder !== importId) throw new HttpError(409, "repository already holds content not written by this import; start a new import");
1307+ }
1308+ }
1309+ if (this.importsModule.hasForeignContent(importId)) throw new HttpError(409, "repository already holds content not written by this import; start a new import");
1310+ this.sql.exec("INSERT OR REPLACE INTO meta (key, value) VALUES ('import_lock', ?)", importId);
1311+ }
1312+
1313+ /** Release the import lock (on finalize, fail or cancel). */
1314+ importEnd(importId: string): void {
1315+ if (this.importLockHolder() === importId) this.sql.exec("DELETE FROM meta WHERE key = 'import_lock'");
1316+ }
1317+
1318+ /** Refuse any non-import write while an import holds the repository. */
1319+ private assertWritable(importId?: string): void {
1320+ const holder = this.importLockHolder();
1321+ if (holder && holder !== importId) throw new HttpError(423, "repository is locked by an import in progress");
1322+ }
1323+
1324+ importProtection(pattern: string, rules: ProtectionRules | null, importId: string) {
1325+ if (rules?.mergeQueue) rules = { ...rules, mergeQueue: normalizeMergeQueue(rules.mergeQueue) };
1326+ if (rules?.requiredCheckSources) rules = { ...rules, requiredCheckSources: normalizeCheckSources(rules.requiredCheckSources) };
1327+ if (rules) this.importsModule.protection(pattern, JSON.stringify(rules), importId);
1328+ }
1329+ importLabels(...a: Parameters<RepoImports["labels"]>) {
1330+ return this.importsModule.labels(...a);
1331+ }
1332+ importMilestones(...a: Parameters<RepoImports["milestones"]>) {
1333+ return this.importsModule.milestones(...a);
1334+ }
1335+ importIssues(...a: Parameters<RepoImports["issues"]>) {
1336+ return this.importsModule.issues(...a);
1337+ }
1338+ importComments(...a: Parameters<RepoImports["comments"]>) {
1339+ return this.importsModule.comments(...a);
1340+ }
1341+ importReviews(...a: Parameters<RepoImports["reviews"]>) {
1342+ return this.importsModule.reviews(...a);
1343+ }
1344+ importSetMergedBy(...a: Parameters<RepoImports["setMergedBy"]>) {
1345+ return this.importsModule.setMergedBy(...a);
1346+ }
1347+ importPullNumbers(...a: Parameters<RepoImports["pullNumbers"]>) {
1348+ return this.importsModule.pullNumbers(...a);
1349+ }
1350+ importCounts() {
1351+ return this.importsModule.counts();
1352+ }
1353+ importFindings() {
1354+ return this.importsModule.findings();
1355+ }
1356+ /**
1357+ * Point `refs/pull/N/head` at each imported pull request's head commit (the server owns
1358+ * refs/pull/*; the agent pushed the commits to refs/diffinery-import/pull/N). Heads whose
1359+ * commits never arrived (e.g. deleted forks) are reported, not fatal.
1360+ */
1361+ importPullRefs(importId?: string): { created: number; missing: number[] } {
1362+ const missing: number[] = [];
1363+ let created = 0;
1364+ for (const p of this.importsModule.pullHeads()) {
1365+ const ref = `refs/pull/${p.number}/head`;
1366+ if (!/^[0-9a-f]{40}$/.test(p.headSha) || !this.store.has(p.headSha)) {
1367+ missing.push(p.number);
1368+ continue;
1369+ }
1370+ const current = this.getRef(ref);
1371+ // Record provenance so a retry does not mistake refs/pull/*/head for content people adopted.
1372+ if (importId) this.importsModule.recordImportRef(ref, p.headSha, importId);
1373+ if (current === p.headSha) continue;
1374+ if (this.casRef(ref, current, p.headSha)) created++;
1375+ }
1376+ return { created, missing };
1377+ }
1378+ /** Remove the import's own refs once refs/pull/N/head keep the pull request commits reachable. */
1379+ importDropTempRefs(importId?: string): number {
1380+ const refs = this.refs(IMPORT_REF_PREFIX);
1381+ for (const r of refs) this.casRef(r.name, r.oid, null);
1382+ if (importId) this.importsModule.dropImportRefs(importId);
1383+ return refs.length;
1384+ }
1385+ setOrgVariables(...a: Parameters<RepoImports["setOrgVariables"]>) {
1386+ return this.importsModule.setOrgVariables(...a);
1387+ }
1388+ issueMilestoneOf(number: number) {
1389+ return this.importsModule.milestoneOf(number);
1390+ }
1391+ /** Check names that workflow jobs on the default branch report, keyed by job name (GitHub names checks by job). */
1392+ async importCheckNames(): Promise<Record<string, string[]>> {
1393+ const sha = await this.resolve("HEAD");
1394+ const out: Record<string, string[]> = {};
1395+ if (!sha) return out;
1396+ for (const w of await this.actions.loadWorkflows(sha)) {
1397+ if (!w.workflow) continue;
1398+ const wf = w.workflow.name ?? w.path.split("/").pop()!;
1399+ for (const [key, job] of Object.entries(w.workflow.jobs)) {
1400+ const name = (job as { name?: unknown }).name;
1401+ for (const n of new Set([key, typeof name === "string" ? name : key])) (out[n] ??= []).push(`${wf} / ${typeof name === "string" && !name.includes("${{") ? name : key}`);
1402+ }
1403+ }
1404+ return out;
1405+ }
1406+ milestones(...a: Parameters<RepoImports["listMilestones"]>) {
1407+ return this.importsModule.listMilestones(...a);
1408+ }
1409+ milestone(...a: Parameters<RepoImports["milestone"]>) {
1410+ return this.importsModule.milestone(...a);
1411+ }
1412+ saveMilestone(...a: Parameters<RepoImports["saveMilestone"]>) {
1413+ return this.importsModule.saveMilestone(...a);
1414+ }
1415+ setIssueMilestone(...a: Parameters<RepoImports["setIssueMilestone"]>) {
1416+ return this.importsModule.setIssueMilestone(...a);
1417+ }
1418+
12471419 // ---------------------------------------------------------------- live updates (WebSockets with hibernation)
12481420
12491421 override async fetch(request: Request): Promise<Response> {
modifiedapps/diffinery/src/repo/actions.ts+6−2View file
@@ -1375,8 +1375,12 @@
13751375 else this.sql.exec("INSERT OR REPLACE INTO variables (name, value, updated_at) VALUES (?, ?, ?)", name.toUpperCase(), value, now());
13761376 }
13771377
1378- variables(): Record<string, string> {
1379- return Object.fromEntries(this.sql.exec<{ name: string; value: string }>("SELECT name, value FROM variables").toArray().map((r) => [r.name, r.value]));
1378+ /** Variables for `vars.*`: the organization's, overridden by the repository's. `withOrg = false` lists only the repository's own. */
1379+ variables(withOrg = true): Record<string, string> {
1380+ const own = Object.fromEntries(this.sql.exec<{ name: string; value: string }>("SELECT name, value FROM variables").toArray().map((r) => [r.name, r.value]));
1381+ if (!withOrg) return own;
1382+ const org = Object.fromEntries(this.sql.exec<{ name: string; value: string }>("SELECT name, value FROM org_variables").toArray().map((r) => [r.name, r.value]));
1383+ return { ...org, ...own };
13801384 }
13811385
13821386 // ---------------------------------------------------------------- cache and artifacts
modifiedapps/diffinery/src/repo/collab.ts+24−2View file
@@ -7,6 +7,22 @@
77 import { COMMENT_SELECT } from "./social.js";
88 import { requiredCheck } from "./required.js";
99
10+type Sig = { name: string; email: string; timestamp: number; timezone: string };
11+
12+/**
13+ * The author signature for a squash or rebase commit. A pull request authored by a GitHub import
14+ * (a mannequin, author_kind "github") is attributed to the original GitHub user through GitHub's
15+ * own noreply domain, never a Diffinery address, so an imported author is never passed off as a
16+ * Diffinery account.
17+ */
18+export function squashSig(author: { login: string; kind?: string | null }, base: Sig): Sig {
19+ if (author.kind === "github") {
20+ const bare = author.login.replace(/^github:/, "");
21+ return { ...base, name: bare, email: `${bare}@users.noreply.github.com` };
22+ }
23+ return { ...base, name: author.login, email: `${author.login}@users.noreply.diffinery.com` };
24+}
25+
1026 export type IssueRow = {
1127 number: number;
1228 kind: "issue" | "pull";
@@ -54,6 +70,7 @@
5470 closedAt: string | null;
5571 comments: number;
5672 locked: boolean;
73+ milestone: { number: number; title: string; state: "open" | "closed"; dueOn: string | null } | null;
5774 pull?: {
5875 head: string;
5976 headSha: string;
@@ -96,6 +113,8 @@
96113 submittedAt: string;
97114 /** GitHub-style numeric id. */
98115 seq?: number;
116+ /** Set when the review was brought in by a GitHub import: it is shown as history, never counted as a vote. */
117+ imported?: boolean;
99118 };
100119
101120 export type Check = {
@@ -179,6 +198,7 @@
179198 closedAt: r.closed_at,
180199 comments,
181200 locked: !!r.locked,
201+ milestone: this.repo.issueMilestoneOf(r.number),
182202 };
183203 if (r.kind === "pull") {
184204 const p = this.pullRow(r.number);
@@ -316,6 +336,7 @@
316336 commitId: r.commit_id!,
317337 submittedAt: r.submitted_at!,
318338 seq: Number(r.seq),
339+ imported: r.imported != null,
319340 }));
320341 }
321342
@@ -713,7 +734,8 @@
713734 result.enforceAdmins = !!rules.enforceAdmins;
714735 result.requiredApprovals = rules.requiredApprovals ?? 0;
715736 const latest = new Map<string, Review>();
716- for (const rv of this.reviews(number)) if (rv.state !== "COMMENTED") latest.set(rv.author.id, rv);
737+ // Imported reviews are history, not votes, whoever authored them; and so are plain comments.
738+ for (const rv of this.reviews(number)) if (rv.state !== "COMMENTED" && rv.author.kind !== "github" && !rv.imported) latest.set(rv.author.id, rv);
717739 for (const rv of latest.values()) {
718740 // Only people's approvals count toward requirements: agents' reviews are advisory, and a
719741 // workflow's token (kind "system", or "agent" for code an agent wrote) never approves its own way in.
@@ -789,7 +811,7 @@
789811 if (merged.conflicts.length || !merged.tree) throw new HttpError(405, "merge conflicts");
790812 const title = input.title ?? (method === "squash" ? `${r.title} (#${number})` : `Merge pull request #${number} from ${p.head_ref}`);
791813 const body = input.message ?? (method === "squash" ? (await this.pullCommits(number)).map((c) => `* ${c.message.split("\n")[0]}`).join("\n") : r.title);
792- const author = method === "squash" ? { ...sig, name: r.author_login, email: `${r.author_login}@users.noreply.diffinery.com` } : sig;
814+ const author = method === "squash" ? squashSig({ login: r.author_login, kind: r.author_kind }, sig) : sig;
793815 newSha = await store.write(
794816 "commit",
795817 serializeCommit({ tree: merged.tree, parents: method === "squash" ? [baseSha] : [baseSha, p.head_sha], author, committer: sig, message: `${title}\n\n${body}\n` }),
addedapps/diffinery/src/repo/imports.ts+373−0View file
@@ -1,0 +1,373 @@
1+import type { ImportedComment, ImportedIssue, ImportedMilestone, ImportedReview } from "../imports/mapping.js";
2+import { HttpError, now } from "../lib/util.js";
3+
4+/** Refs only a GitHub import writes (pull request heads and chunks in flight); see containers/runner/src/importer.ts. */
5+export const IMPORT_REF_PREFIX = "refs/diffinery-import/";
6+
7+export type Milestone = {
8+ number: number;
9+ title: string;
10+ description: string;
11+ state: "open" | "closed";
12+ dueOn: string | null;
13+ createdAt: string;
14+ updatedAt: string;
15+ closedAt: string | null;
16+ creator: string | null;
17+ openIssues: number;
18+ closedIssues: number;
19+};
20+
21+type MilestoneRow = {
22+ number: number;
23+ title: string;
24+ description: string;
25+ state: "open" | "closed";
26+ due_on: string | null;
27+ created_at: string;
28+ updated_at: string;
29+ closed_at: string | null;
30+ creator: string | null;
31+};
32+
33+/**
34+ * Repository side of GitHub imports, plus milestones. Imported rows keep their
35+ * GitHub numbers, timestamps and people, and every write is an upsert keyed by
36+ * the GitHub id, so a retried import step never duplicates anything. Nothing
37+ * here emits events: an import fires no webhooks, workflows or notifications.
38+ */
39+export class RepoImports {
40+ constructor(private readonly sql: SqlStorage) {}
41+
42+ // ---------------------------------------------------------------- labels and milestones
43+
44+ labels(list: Array<{ name: string; color: string; description: string }>, importId: string): number {
45+ for (const l of list)
46+ this.sql.exec(
47+ `INSERT INTO labels (name, color, description, imported) VALUES (?, ?, ?, ?)
48+ ON CONFLICT(name) DO UPDATE SET color = excluded.color, description = excluded.description, imported = excluded.imported
49+ WHERE labels.imported = excluded.imported`,
50+ l.name,
51+ (l.color || "888888").replace(/^#/, ""),
52+ l.description ?? "",
53+ importId,
54+ );
55+ return list.length;
56+ }
57+
58+ milestones(list: ImportedMilestone[], importId: string): number {
59+ for (const m of list) {
60+ this.sql.exec(
61+ `INSERT INTO milestones (number, title, description, state, due_on, created_at, updated_at, closed_at, creator, imported)
62+ VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
63+ ON CONFLICT(number) DO UPDATE SET title = excluded.title, description = excluded.description, state = excluded.state,
64+ due_on = excluded.due_on, created_at = excluded.created_at, updated_at = excluded.updated_at, closed_at = excluded.closed_at,
65+ creator = excluded.creator, imported = excluded.imported
66+ WHERE milestones.imported = excluded.imported`,
67+ m.number,
68+ m.title,
69+ m.description,
70+ m.state,
71+ m.dueOn,
72+ m.createdAt,
73+ m.updatedAt,
74+ m.closedAt,
75+ m.creator,
76+ importId,
77+ );
78+ }
79+ this.raiseCounter("milestone", Math.max(0, ...list.map((m) => m.number)));
80+ return list.length;
81+ }
82+
83+ // ---------------------------------------------------------------- issues and pull requests
84+
85+ /** Insert issues and pull requests with their original numbers; only overwrites rows this import owns. */
86+ issues(list: ImportedIssue[], importId: string): number {
87+ for (const i of list) {
88+ this.sql.exec(
89+ `INSERT INTO issues (number, kind, title, body, state, state_reason, author_id, author_login, author_kind, assignees, labels, created_at, updated_at, closed_at, locked, imported)
90+ VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
91+ ON CONFLICT(number) DO UPDATE SET kind = excluded.kind, title = excluded.title, body = excluded.body, state = excluded.state,
92+ state_reason = excluded.state_reason, author_id = excluded.author_id, author_login = excluded.author_login, author_kind = excluded.author_kind,
93+ assignees = excluded.assignees, labels = excluded.labels, created_at = excluded.created_at, updated_at = excluded.updated_at,
94+ closed_at = excluded.closed_at, locked = excluded.locked, imported = excluded.imported
95+ WHERE issues.imported = excluded.imported`,
96+ i.number,
97+ i.kind,
98+ i.title,
99+ i.body,
100+ i.state,
101+ i.stateReason,
102+ i.author.id,
103+ i.author.login,
104+ i.author.kind,
105+ JSON.stringify(i.assignees),
106+ JSON.stringify(i.labels),
107+ i.createdAt,
108+ i.updatedAt,
109+ i.closedAt,
110+ i.locked ? 1 : 0,
111+ importId,
112+ );
113+ if (i.pull) {
114+ const p = i.pull;
115+ this.sql.exec(
116+ `INSERT INTO pulls (number, head_ref, head_sha, base_ref, draft, merged, merged_at, merged_by, merge_commit, requested_reviewers, imported)
117+ VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
118+ ON CONFLICT(number) DO UPDATE SET head_ref = excluded.head_ref, head_sha = excluded.head_sha, base_ref = excluded.base_ref, draft = excluded.draft,
119+ merged = excluded.merged, merged_at = excluded.merged_at, merged_by = COALESCE(excluded.merged_by, pulls.merged_by),
120+ merge_commit = excluded.merge_commit, requested_reviewers = excluded.requested_reviewers, imported = excluded.imported
121+ WHERE pulls.imported = excluded.imported`,
122+ i.number,
123+ p.headRef,
124+ p.headSha,
125+ p.baseRef,
126+ p.draft ? 1 : 0,
127+ p.merged ? 1 : 0,
128+ p.mergedAt,
129+ p.mergedBy,
130+ p.mergeCommit,
131+ JSON.stringify(p.requestedReviewers),
132+ importId,
133+ );
134+ }
135+ if (i.milestone) this.sql.exec("INSERT OR REPLACE INTO issue_milestones (number, milestone) VALUES (?, ?)", i.number, i.milestone);
136+ else this.sql.exec("DELETE FROM issue_milestones WHERE number = ?", i.number);
137+ }
138+ this.raiseCounter("issue", Math.max(0, ...list.map((i) => i.number)));
139+ return list.length;
140+ }
141+
142+ /** GitHub reports who merged a pull request only on the single pull request. */
143+ setMergedBy(number: number, login: string): void {
144+ this.sql.exec("UPDATE pulls SET merged_by = ? WHERE number = ? AND merged = 1", login, number);
145+ }
146+
147+ /** Pull request numbers after `after`, for the reviews pass. */
148+ pullNumbers(after: number, limit: number): Array<{ number: number; merged: boolean }> {
149+ return this.sql
150+ .exec<{ number: number; merged: number }>("SELECT number, merged FROM pulls WHERE number > ? ORDER BY number LIMIT ?", after, limit)
151+ .toArray()
152+ .map((r) => ({ number: r.number, merged: !!r.merged }));
153+ }
154+
155+ comments(list: ImportedComment[], importId: string): number {
156+ let n = 0;
157+ for (const c of list) {
158+ if (!this.sql.exec<{ n: number }>("SELECT count(*) AS n FROM issues WHERE number = ?", c.number).one().n) continue;
159+ // Not INSERT OR REPLACE: that reassigns the rowid, and comment "seq" (the public id) is the rowid.
160+ this.sql.exec(
161+ `INSERT INTO comments (id, number, author_id, author_login, author_kind, body, created_at, updated_at, review_id, path, line, side, commit_id, in_reply_to, imported)
162+ VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
163+ ON CONFLICT(id) DO UPDATE SET number = excluded.number, author_id = excluded.author_id, author_login = excluded.author_login,
164+ author_kind = excluded.author_kind, body = excluded.body, created_at = excluded.created_at, updated_at = excluded.updated_at,
165+ review_id = excluded.review_id, path = excluded.path, line = excluded.line, side = excluded.side, commit_id = excluded.commit_id,
166+ in_reply_to = excluded.in_reply_to, imported = excluded.imported
167+ WHERE comments.imported = excluded.imported`,
168+ c.id,
169+ c.number,
170+ c.author.id,
171+ c.author.login,
172+ c.author.kind,
173+ c.body,
174+ c.createdAt,
175+ c.updatedAt,
176+ c.reviewId ?? null,
177+ c.path ?? null,
178+ c.line ?? null,
179+ c.side ?? null,
180+ c.commitId ?? null,
181+ c.inReplyTo ?? null,
182+ importId,
183+ );
184+ n++;
185+ }
186+ return n;
187+ }
188+
189+ reviews(list: ImportedReview[], importId: string): number {
190+ for (const r of list) {
191+ this.sql.exec(
192+ `INSERT INTO reviews (id, number, author_id, author_login, author_kind, state, body, commit_id, submitted_at, imported)
193+ VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
194+ ON CONFLICT(id) DO UPDATE SET number = excluded.number, author_id = excluded.author_id, author_login = excluded.author_login,
195+ author_kind = excluded.author_kind, state = excluded.state, body = excluded.body, commit_id = excluded.commit_id,
196+ submitted_at = excluded.submitted_at, imported = excluded.imported
197+ WHERE reviews.imported = excluded.imported`,
198+ r.id,
199+ r.number,
200+ r.author.id,
201+ r.author.login,
202+ r.author.kind,
203+ r.state,
204+ r.body,
205+ r.commitId,
206+ r.submittedAt,
207+ importId,
208+ );
209+ }
210+ return list.length;
211+ }
212+
213+ /** New issues continue after the highest imported number. */
214+ private raiseCounter(name: string, value: number): void {
215+ if (value > 0) this.sql.exec("INSERT INTO counters (name, value) VALUES (?, ?) ON CONFLICT(name) DO UPDATE SET value = max(value, excluded.value)", name, value);
216+ }
217+
218+ counts(): { issues: number; pulls: number; comments: number; reviews: number; labels: number; milestones: number; maxNumber: number } {
219+ const q = (s: string) => this.sql.exec<{ n: number }>(s).one().n;
220+ return {
221+ issues: q("SELECT count(*) AS n FROM issues WHERE kind = 'issue'"),
222+ pulls: q("SELECT count(*) AS n FROM issues WHERE kind = 'pull'"),
223+ comments: q("SELECT count(*) AS n FROM comments"),
224+ reviews: q("SELECT count(*) AS n FROM reviews"),
225+ labels: q("SELECT count(*) AS n FROM labels"),
226+ milestones: q("SELECT count(*) AS n FROM milestones"),
227+ maxNumber: q("SELECT COALESCE(max(number), 0) AS n FROM issues"),
228+ };
229+ }
230+
231+ // ---------------------------------------------------------------- push protection findings
232+
233+ /** Secrets an import pushed with an admin bypass, so the summary can list what to rotate. */
234+ recordFindings(leaks: Array<{ oid: string; kind: string }>): void {
235+ for (const l of leaks.slice(0, 500)) this.sql.exec("INSERT OR IGNORE INTO import_findings (blob, kind, at) VALUES (?, ?, ?)", l.oid, l.kind, now());
236+ }
237+
238+ findings(): Array<{ blob: string; kind: string; at: string }> {
239+ return this.sql.exec<{ blob: string; kind: string; at: string }>("SELECT blob, kind, at FROM import_findings ORDER BY at, blob").toArray();
240+ }
241+
242+ /** Imported pull requests and the head commit each one pins. */
243+ pullHeads(): Array<{ number: number; headSha: string }> {
244+ return this.sql.exec<{ number: number; head_sha: string }>("SELECT number, head_sha FROM pulls ORDER BY number").toArray().map((r) => ({ number: r.number, headSha: r.head_sha }));
245+ }
246+
247+ // ---------------------------------------------------------------- import lock and provenance (H2)
248+
249+ /**
250+ * True when the repository already holds any issue, pull request, comment, review, label,
251+ * milestone or protection that this import did not write. A retry refuses to reuse a repo
252+ * that people have since adopted, so an import never clobbers real content.
253+ */
254+ hasForeignContent(importId: string): boolean {
255+ const tables = ["issues", "pulls", "comments", "reviews", "labels", "milestones", "protections"];
256+ for (const t of tables) {
257+ const n = this.sql.exec<{ n: number }>(`SELECT count(*) AS n FROM ${t} WHERE imported IS NULL OR imported <> ?`, importId).one().n;
258+ if (n > 0) return true;
259+ }
260+ // A milestone assignment or issue-milestone link is content too; covered by issues/milestones above.
261+ return false;
262+ }
263+
264+ /** Record that this import set `ref` to `oid` (so a later update may move only refs it owns). */
265+ recordImportRef(ref: string, oid: string, importId: string): void {
266+ this.sql.exec("INSERT INTO import_refs (ref, oid, import_id) VALUES (?, ?, ?) ON CONFLICT(ref) DO UPDATE SET oid = excluded.oid, import_id = excluded.import_id", ref, oid, importId);
267+ }
268+
269+ /** The oid this import last wrote to `ref`, or null if the import never wrote it. */
270+ importRefOid(ref: string, importId: string): string | null {
271+ const r = this.sql.exec<{ oid: string }>("SELECT oid FROM import_refs WHERE ref = ? AND import_id = ?", ref, importId).toArray()[0];
272+ return r ? r.oid : null;
273+ }
274+
275+ dropImportRefs(importId: string): void {
276+ this.sql.exec("DELETE FROM import_refs WHERE import_id = ?", importId);
277+ }
278+
279+ /** Write imported branch protection, only overwriting a rule this import owns. */
280+ protection(pattern: string, rules: string, importId: string): void {
281+ this.sql.exec(
282+ `INSERT INTO protections (pattern, rules, imported) VALUES (?, ?, ?)
283+ ON CONFLICT(pattern) DO UPDATE SET rules = excluded.rules, imported = excluded.imported
284+ WHERE protections.imported = excluded.imported`,
285+ pattern,
286+ rules,
287+ importId,
288+ );
289+ }
290+
291+ // ---------------------------------------------------------------- organization variables
292+
293+ /** Replace the copy of the owning organization's Actions variables. */
294+ setOrgVariables(vars: Record<string, string>): void {
295+ this.sql.exec("DELETE FROM org_variables");
296+ for (const [name, value] of Object.entries(vars)) this.sql.exec("INSERT INTO org_variables (name, value) VALUES (?, ?)", name, value);
297+ }
298+
299+ // ---------------------------------------------------------------- milestones
300+
301+ private milestoneRow(m: MilestoneRow): Milestone {
302+ const counts = this.sql
303+ .exec<{ state: string; n: number }>("SELECT i.state, count(*) AS n FROM issue_milestones x JOIN issues i ON i.number = x.number WHERE x.milestone = ? GROUP BY i.state", m.number)
304+ .toArray();
305+ return {
306+ number: m.number,
307+ title: m.title,
308+ description: m.description,
309+ state: m.state,
310+ dueOn: m.due_on,
311+ createdAt: m.created_at,
312+ updatedAt: m.updated_at,
313+ closedAt: m.closed_at,
314+ creator: m.creator,
315+ openIssues: counts.find((c) => c.state === "open")?.n ?? 0,
316+ closedIssues: counts.find((c) => c.state === "closed")?.n ?? 0,
317+ };
318+ }
319+
320+ listMilestones(state: "open" | "closed" | "all" = "open"): Milestone[] {
321+ const rows = this.sql
322+ .exec<MilestoneRow>(`SELECT * FROM milestones ${state === "all" ? "" : "WHERE state = ?"} ORDER BY COALESCE(due_on, '9999'), number`, ...(state === "all" ? [] : [state]))
323+ .toArray();
324+ return rows.map((r) => this.milestoneRow(r));
325+ }
326+
327+ milestone(number: number): Milestone {
328+ const r = this.sql.exec<MilestoneRow>("SELECT * FROM milestones WHERE number = ?", number).toArray()[0];
329+ if (!r) throw new HttpError(404, "milestone not found");
330+ return this.milestoneRow(r);
331+ }
332+
333+ saveMilestone(number: number | null, input: { title?: string; description?: string; state?: "open" | "closed"; dueOn?: string | null }, creator: string): Milestone {
334+ const t = now();
335+ if (number === null) {
336+ if (!input.title?.trim()) throw new HttpError(422, "title is required");
337+ this.sql.exec("INSERT INTO counters (name, value) VALUES ('milestone', 1) ON CONFLICT(name) DO UPDATE SET value = value + 1");
338+ const n = this.sql.exec<{ value: number }>("SELECT value FROM counters WHERE name = 'milestone'").one().value;
339+ this.sql.exec("INSERT INTO milestones (number, title, description, state, due_on, created_at, updated_at, creator) VALUES (?, ?, ?, 'open', ?, ?, ?, ?)", n, input.title.trim(), input.description ?? "", input.dueOn ?? null, t, t, creator);
340+ return this.milestone(n);
341+ }
342+ const m = this.milestone(number);
343+ const state = input.state ?? m.state;
344+ this.sql.exec(
345+ "UPDATE milestones SET title = ?, description = ?, state = ?, due_on = ?, updated_at = ?, closed_at = ? WHERE number = ?",
346+ input.title?.trim() || m.title,
347+ input.description ?? m.description,
348+ state,
349+ input.dueOn === undefined ? m.dueOn : input.dueOn,
350+ t,
351+ state === "closed" ? (m.closedAt ?? t) : null,
352+ number,
353+ );
354+ return this.milestone(number);
355+ }
356+
357+ setIssueMilestone(number: number, milestone: number | null): void {
358+ if (!this.sql.exec<{ n: number }>("SELECT count(*) AS n FROM issues WHERE number = ?", number).one().n) throw new HttpError(404, `#${number} not found`);
359+ if (milestone === null) {
360+ this.sql.exec("DELETE FROM issue_milestones WHERE number = ?", number);
361+ return;
362+ }
363+ this.milestone(milestone);
364+ this.sql.exec("INSERT OR REPLACE INTO issue_milestones (number, milestone) VALUES (?, ?)", number, milestone);
365+ }
366+
367+ milestoneOf(number: number): { number: number; title: string; state: "open" | "closed"; dueOn: string | null } | null {
368+ const r = this.sql
369+ .exec<{ number: number; title: string; state: "open" | "closed"; due_on: string | null }>("SELECT m.number, m.title, m.state, m.due_on FROM issue_milestones x JOIN milestones m ON m.number = x.milestone WHERE x.number = ?", number)
370+ .toArray()[0];
371+ return r ? { number: r.number, title: r.title, state: r.state, dueOn: r.due_on } : null;
372+ }
373+}
modifiedapps/diffinery/src/repo/queue.ts+2−1View file
@@ -1,4 +1,5 @@
11 import { ZERO_OID, mergeBase, mergeTrees, readCommit, serializeCommit } from "@diffinery/git-core";
2+import { squashSig } from "./collab.js";
23 import { requiredCheck } from "./required.js";
34 import type { Actor } from "@diffinery/shared";
45 import { HttpError, now } from "../lib/util.js";
@@ -338,7 +339,7 @@
338339 r.method === "squash"
339340 ? `${issue.title} (#${r.number})\n\n${(await this.repo.collab.pullCommits(r.number)).map((c) => `* ${c.message.split("\n")[0]}`).join("\n")}\n`
340341 : `Merge pull request #${r.number} from ${pull.head}\n\n${issue.title}\n`;
341- const author = r.method === "squash" ? { ...committer, name: issue.author.login, email: `${issue.author.login}@users.noreply.diffinery.com` } : committer;
342+ const author = r.method === "squash" ? squashSig(issue.author, committer) : committer;
342343 sha = await store.write("commit", serializeCommit({ tree: merged.tree, parents: r.method === "squash" ? [parent] : [parent, r.head_sha], author, committer, message }));
343344 }
344345 const ref = `refs/heads/${QUEUE_BRANCH_PREFIX}${pull.base}/pr-${r.number}-${r.head_sha}`;
modifiedapps/diffinery/src/repo/schema.ts+20−0View file
@@ -134,6 +134,18 @@
134134 -- Who last moved each ref: tags made by agents never match environment tag patterns
135135 -- (./deployments.ts), and runs of code an agent pushed act as the agent (./token-policy.ts).
136136 CREATE TABLE IF NOT EXISTS ref_actors (ref TEXT PRIMARY KEY, actor_id TEXT NOT NULL, actor_kind TEXT NOT NULL, updated_at TEXT NOT NULL);
137+-- GitHub imports and milestones (see ./imports.ts).
138+CREATE TABLE IF NOT EXISTS milestones (
139+ number INTEGER PRIMARY KEY, title TEXT NOT NULL, description TEXT NOT NULL DEFAULT '', state TEXT NOT NULL DEFAULT 'open',
140+ due_on TEXT, created_at TEXT NOT NULL, updated_at TEXT NOT NULL, closed_at TEXT, creator TEXT
141+);
142+CREATE TABLE IF NOT EXISTS issue_milestones (number INTEGER PRIMARY KEY, milestone INTEGER NOT NULL);
143+CREATE INDEX IF NOT EXISTS issue_milestones_milestone ON issue_milestones(milestone);
144+CREATE TABLE IF NOT EXISTS import_findings (blob TEXT NOT NULL, kind TEXT NOT NULL, at TEXT NOT NULL, PRIMARY KEY (blob, kind));
145+-- Refs a running import wrote, with the oid it set, so a retry only moves refs it owns (never a ref people adopted).
146+CREATE TABLE IF NOT EXISTS import_refs (ref TEXT PRIMARY KEY, oid TEXT NOT NULL, import_id TEXT NOT NULL);
147+-- A copy of the owning organization's Actions variables (D1 org_variables), kept in sync by the Directory.
148+CREATE TABLE IF NOT EXISTS org_variables (name TEXT PRIMARY KEY, value TEXT NOT NULL);
137149 `;
138150
139151 /** Columns added after a table was first created; added with ALTER TABLE when missing. */
@@ -160,6 +172,14 @@
160172 ["jobs", "continue_on_error", "INTEGER NOT NULL DEFAULT 0"],
161173 ["jobs", "concurrency_group", "TEXT"],
162174 ["jobs", "concurrency_cancel", "INTEGER NOT NULL DEFAULT 0"],
175+ // The GitHub import that wrote this row (null for everything else); a retry may only overwrite its own rows.
176+ ["issues", "imported", "TEXT"],
177+ ["pulls", "imported", "TEXT"],
178+ ["comments", "imported", "TEXT"],
179+ ["reviews", "imported", "TEXT"],
180+ ["labels", "imported", "TEXT"],
181+ ["milestones", "imported", "TEXT"],
182+ ["protections", "imported", "TEXT"],
163183 ];
164184
165185 export function migrateRepoSchema(sql: SqlStorage): void {
modifiedapps/diffinery/src/repo/search.ts+16−4View file
@@ -47,13 +47,25 @@
4747 if (q.merged !== undefined) where.push(q.merged ? "p.merged = 1" : "coalesce(p.merged, 0) = 0");
4848 if (q.draft !== undefined) where.push(q.draft ? "p.draft = 1" : "coalesce(p.draft, 0) = 0");
4949 if (q.locked !== undefined) where.push(q.locked ? "i.locked = 1" : "i.locked = 0");
50+ // Author search distinguishes imported GitHub users from Diffinery accounts: "author:github:alice"
51+ // matches the mannequin, a plain "author:alice" matches only the real Diffinery account.
5052 if (q.author) {
51- where.push("i.author_login = ? COLLATE NOCASE");
52- args.push(q.author);
53+ if (q.author.startsWith("github:")) {
54+ where.push("i.author_kind = 'github' AND i.author_login = ? COLLATE NOCASE");
55+ args.push(q.author.slice(7));
56+ } else {
57+ where.push("i.author_login = ? COLLATE NOCASE AND i.author_kind != 'github'");
58+ args.push(q.author);
59+ }
5360 }
5461 if (q.notAuthor) {
55- where.push("i.author_login != ? COLLATE NOCASE");
56- args.push(q.notAuthor);
62+ if (q.notAuthor.startsWith("github:")) {
63+ where.push("NOT (i.author_kind = 'github' AND i.author_login = ? COLLATE NOCASE)");
64+ args.push(q.notAuthor.slice(7));
65+ } else {
66+ where.push("NOT (i.author_login = ? COLLATE NOCASE AND i.author_kind != 'github')");
67+ args.push(q.notAuthor);
68+ }
5769 }
5870 if (q.head) {
5971 where.push("p.head_ref = ?");
modifiedapps/diffinery/src/repo/types.ts+2−0View file
@@ -9,6 +9,8 @@
99 role: Role;
1010 /** A workflow job token's effective `permissions:` (lib/job-permissions.ts); absent for everyone else. */
1111 permissions?: Record<string, "none" | "read" | "write">;
12+ /** Set only for a running import's own push token, after the Worker verified it (see http/git.ts). */
13+ importId?: string;
1214 }
1315
1416 export interface RepoMeta {
modifiedapps/diffinery/src/web/account.tsx+18−5View file
@@ -329,7 +329,9 @@
329329 const NewRepoForm: FC<{ t: string; me: Account; orgs: Array<Account & { role: string }>; values: Record<string, string>; error?: string }> = ({ t, me, orgs, values, error }) => (
330330 <div style="max-width:720px;margin:0 auto">
331331 <h1>Create a new repository</h1>
332- <p class="muted">A repository holds your project's files, history, issues, pull requests and workflows.</p>
332+ <p class="muted">
333+ A repository holds your project's files, history, issues, pull requests and workflows. Already on GitHub? <a href="/new/import">Import a repository</a>.
334+ </p>
333335 {error ? <div class="flash error">{error}</div> : null}
334336 <Form action="/new" t={t} class="box">
335337 <div class="body">
@@ -583,7 +585,7 @@
583585
584586 // ---------------------------------------------------------------- /settings
585587
586-export type SettingsTab = "profile" | "appearance" | "security" | "tokens" | "agents" | "orgs" | "notifications" | "admin" | "email" | "audit";
588+export type SettingsTab = "profile" | "appearance" | "security" | "tokens" | "agents" | "orgs" | "notifications" | "linked" | "admin" | "email" | "audit";
587589
588590 export const SettingsNav: FC<{ me: Account; tab: SettingsTab }> = ({ me, tab }) => (
589591 <aside class="side vt-side">
@@ -617,6 +619,10 @@
617619 <a href="/settings/notifications" class={tab === "notifications" ? "on" : ""}>
618620 Notifications
619621 </a>
622+ <a href="/settings/linked-accounts" class={tab === "linked" ? "on" : ""}>
623+ Linked accounts
624+ </a>
625+ <a href="/imports">Imports</a>
620626 {me.site_admin ? (
621627 <>
622628 <h4>Site administration</h4>
@@ -1661,9 +1667,16 @@
16611667 <div class="row between mb">
16621668 <h2 class="m0">Repositories</h2>
16631669 {role ? (
1664- <a class="btn sm primary" href={`/new?owner=${encodeURIComponent(acct.login)}`}>
1665- {raw(ICONS.plus)} New repository
1666- </a>
1670+ <span class="row" style="gap:6px">
1671+ {isAdmin ? (
1672+ <a class="btn sm" href={`/new/import?owner=${encodeURIComponent(acct.login)}&mode=owner`}>
1673+ Import from GitHub
1674+ </a>
1675+ ) : null}
1676+ <a class="btn sm primary" href={`/new?owner=${encodeURIComponent(acct.login)}`}>
1677+ {raw(ICONS.plus)} New repository
1678+ </a>
1679+ </span>
16671680 ) : null}
16681681 </div>
16691682 <div class="box">
modifiedapps/diffinery/src/web/collab.tsx+45−2View file
@@ -253,6 +253,7 @@
253253 behalf: Map<string, string>;
254254 /** The signed-in person's thread subscription (explicit choice, if any) and repository watch level. */
255255 sub?: { explicit: boolean | null; watch: WatchLevel };
256+ milestones: Array<{ number: number; title: string; state: "open" | "closed" }>;
256257 }
257258
258259 const CommentCard: FC<{ x: ConvCtx; author: { login: string; kind: string }; at: string; body: string; verb?: string; id?: string; extra?: Child; cls?: string }> = ({ x, author, at, body, verb, id, extra, cls }) => (
@@ -515,6 +516,38 @@
515516 ) : null}
516517 </section>
517518 <section>
519+ <h3>Milestone</h3>
520+ {x.issue.milestone ? (
521+ <p class="small m0">
522+ <a href={`${base}/issues${qs({ q: `milestone:${x.issue.milestone.number}`, state: "all" })}`}>{x.issue.milestone.title}</a>{" "}
523+ {x.issue.milestone.state === "closed" ? <span class="badge">closed</span> : null}
524+ {x.issue.milestone.dueOn ? <span class="muted"> · due {x.issue.milestone.dueOn.slice(0, 10)}</span> : null}
525+ </p>
526+ ) : (
527+ <p class="muted small m0">No milestone</p>
528+ )}
529+ {canTriage && (x.milestones.length || x.issue.milestone) ? (
530+ <details style="margin-top:8px">
531+ <summary class="small" style="cursor:pointer;color:var(--accent)">
532+ Set milestone
533+ </summary>
534+ <Form action={`${base}/issues/${x.issue.number}/milestone`} t={t} style="margin-top:8px">
535+ <KSelect
536+ name="milestone"
537+ block
538+ size="sm"
539+ label="Milestone"
540+ value={x.issue.milestone ? String(x.issue.milestone.number) : ""}
541+ options={[{ value: "", label: "No milestone" }, ...x.milestones.map((m) => ({ value: String(m.number), label: m.title }))]}
542+ />
543+ <button class="btn sm" style="margin-top:6px">
544+ Save
545+ </button>
546+ </Form>
547+ </details>
548+ ) : null}
549+ </section>
550+ <section>
518551 <h3>Assignees</h3>
519552 {x.issue.assignees.length ? (
520553 x.issue.assignees.map((a) => (
@@ -915,7 +948,7 @@
915948 async function loadConv(c: AppContext): Promise<ConvCtx> {
916949 const l = await loadRepo(c, prm(c, "owner"), prm(c, "repo"), "read");
917950 const n = issueNumber(c);
918- const [data, labels] = await Promise.all([l.stub.issue(n), l.stub.labels()]);
951+ const [data, labels, milestones] = await Promise.all([l.stub.issue(n), l.stub.labels(), l.stub.milestones("open")]);
919952 const behalf = await onBehalfOf(c, [data.issue.author, ...data.comments.map((x) => x.author), ...data.reviews.map((x) => x.author)]);
920953 const me = principalAccount(c);
921954 let sub: ConvCtx["sub"];
@@ -924,7 +957,7 @@
924957 const [explicit, watch] = await Promise.all([store.threadSub(me.id, l.repo.id, n), store.watchLevel(me.id, l.repo.id)]);
925958 sub = { explicit, watch };
926959 }
927- return { c, l, issue: data.issue, comments: data.comments, reviews: data.reviews, labels, behalf, sub };
960+ return { c, l, issue: data.issue, comments: data.comments, reviews: data.reviews, labels, behalf, sub, milestones };
928961 }
929962
930963 async function issuePage(c: AppContext): Promise<Response> {
@@ -1246,6 +1279,16 @@
12461279 });
12471280 });
12481281
1282+ post(web, "/:owner/:repo/issues/:n/milestone", async (c, f) => {
1283+ const l = await forAction(c, "triage");
1284+ const n = issueNumber(c);
1285+ const back = await backTo(l, n);
1286+ return act(c, back, async () => {
1287+ const m = f.s("milestone");
1288+ await l.stub.setIssueMilestone(n, m ? Number(m) : null);
1289+ });
1290+ });
1291+
12491292 post(web, "/:owner/:repo/pull/:n/review", async (c, f) => {
12501293 const l = await forAction(c);
12511294 const n = issueNumber(c);
addedapps/diffinery/src/web/imports.tsx+644−0View file
@@ -1,0 +1,644 @@
1+/** @jsxImportSource hono/jsx */
2+import { raw } from "hono/html";
3+import type { Child, FC } from "hono/jsx";
4+import { requireAccount, type AppContext } from "../auth/principal.js";
5+import type { Account } from "../lib/directory.js";
6+import { loadRepo } from "../http/common.js";
7+import { ImportService, type Candidate, type ImportRow, type ImportSummary, type StepState, type StoredRequest } from "../imports/service.js";
8+import { webHostOf } from "../imports/options.js";
9+import { HttpError } from "../lib/util.js";
10+import { SettingsNav } from "./account.js";
11+import { csrfToken } from "./csrf.js";
12+import { KCheckbox, KRadio, KSelect } from "./kumo.js";
13+import { Actor, Form, Status, Time, act, fmtBytes, flash, post, prm, qs, render, repoUrl, type WebApp } from "./ui.js";
14+
15+/**
16+ * Import from GitHub (Kumo controls throughout): the start form (one repository,
17+ * or many from an organization or user), the repository picker for bulk imports,
18+ * auto-refreshing progress pages, the list of past imports, the page for setting
19+ * secret values afterwards, and the linked-accounts settings page.
20+ */
21+
22+const OPTIONS: Array<[keyof StoredRequest["options"], string, string]> = [
23+ ["issues", "Issues and pull requests", "With comments, reviews, labels and milestones, keeping their numbers."],
24+ ["labels", "Labels and milestones", "Even without issues."],
25+ ["protection", "Branch protection", "Mapped to Diffinery's rules; required checks are matched to this repository's workflows."],
26+ ["variables", "Actions variables and secret names", "Variable values and the names of secrets (never their values)."],
27+ ["releases", "Releases", "When Diffinery supports releases; until then their tags are imported with the Git data."],
28+ ["lfs", "Git LFS objects", "Large files tracked with Git LFS."],
29+];
30+
31+const statusBadge = (s: string) =>
32+ s === "succeeded" ? <span class="badge green">succeeded</span> : s === "failed" ? <span class="badge red">failed</span> : s === "cancelled" ? <span class="badge">cancelled</span> : s === "running" ? <span class="badge amber">running</span> : s === "draft" ? <span class="badge">draft</span> : <span class="badge">{s}</span>;
33+
34+const StartForm: FC<{ t: string; owners: string[]; values: Record<string, string>; error?: string }> = ({ t, owners, values, error }) => (
35+ <div style="max-width:760px;margin:0 auto">
36+ <h1>Import from GitHub</h1>
37+ <p class="muted">Copy a repository — or every repository of an organization or user — from GitHub.com or GitHub Enterprise, keeping its history, issues, pull requests, reviews, labels and more.</p>
38+ {error ? <div class="flash error">{error}</div> : null}
39+ <Form action="/new/import" t={t} class="box">
40+ <div class="body">
41+ <div class="field">
42+ <label>What to import</label>
43+ <KRadio
44+ name="mode"
45+ label="What to import"
46+ value={values.mode === "owner" ? "owner" : "repo"}
47+ items={[
48+ { value: "repo", label: "One repository", description: "owner/name, or a GitHub URL." },
49+ { value: "owner", label: "A whole organization or user", description: "Choose which of its repositories to import on the next step." },
50+ ]}
51+ />
52+ </div>
53+ <div class="field">
54+ <label for="source">GitHub repository, or organization/user</label>
55+ <input id="source" name="source" type="text" value={values.source} placeholder="octocat/hello-world or octocat" autofocus autocomplete="off" required />
56+ <p class="muted small m0">Example: a URL like https://github.com/octocat/hello-world, or just octocat/hello-world. For a whole account, its name.</p>
57+ </div>
58+ <div class="row" style="align-items:flex-end;gap:10px">
59+ <div class="field" style="min-width:200px">
60+ <label for="owner">Import into</label>
61+ <KSelect id="owner" name="owner" block label="Import into" value={values.owner && owners.includes(values.owner) ? values.owner : (owners[0] ?? "")} options={owners.map((o) => ({ value: o, label: o }))} />
62+ </div>
63+ <div class="field grow">
64+ <label for="name">
65+ New repository name <span class="faint">(one repository only; defaults to the GitHub name)</span>
66+ </label>
67+ <input id="name" name="name" type="text" value={values.name} autocomplete="off" placeholder="hello-world" />
68+ </div>
69+ </div>
70+ <div class="field">
71+ <label>Visibility</label>
72+ <KRadio
73+ name="visibility"
74+ label="Visibility"
75+ value={values.visibility || "keep"}
76+ orientation="horizontal"
77+ items={[
78+ { value: "keep", label: "Keep" },
79+ { value: "private", label: "Private" },
80+ { value: "public", label: "Public" },
81+ ]}
82+ />
83+ </div>
84+ <div class="field">
85+ <label for="github_token">
86+ GitHub token <span class="faint">(required for private repositories, LFS, variables and protection)</span>
87+ </label>
88+ <input id="github_token" name="github_token" type="password" autocomplete="off" placeholder="ghp_… or a fine-grained token" value={values.github_token} />
89+ <p class="muted small m0">A read token (classic `repo`, or a fine-grained token with Contents, Issues, Pull requests, Metadata). Used only for this import, encrypted at rest, and deleted when the import finishes.</p>
90+ </div>
91+ <details>
92+ <summary class="small" style="cursor:pointer;color:var(--accent)">
93+ What to copy, and GitHub Enterprise
94+ </summary>
95+ <div class="kstack mt" style="margin-top:8px">
96+ {OPTIONS.map(([k, label, desc]) => (
97+ <KCheckbox name={`opt_${k}`} checked={values[`opt_${k}`] !== "0"} label={<span><b>{label}</b> <span class="muted small">{desc}</span></span>} />
98+ ))}
99+ </div>
100+ <div class="field mt" style="margin-top:12px">
101+ <label>Secrets found in old history (push protection)</label>
102+ <KRadio
103+ name="secrets"
104+ label="Secrets in history"
105+ value={values.secrets || "bypass"}
106+ items={[
107+ { value: "bypass", label: "Import with an audited bypass", description: "The history is already on GitHub. A human admin's bypass is recorded, and the summary lists what to rotate." },
108+ { value: "stop", label: "Stop the import", description: "Fail if push protection finds a secret, so you can scrub the history first." },
109+ ]}
110+ />
111+ </div>
112+ <div class="field mt" style="margin-top:12px">
113+ <label for="github_api_url">
114+ GitHub Enterprise API URL <span class="faint">(optional)</span>
115+ </label>
116+ <input id="github_api_url" name="github_api_url" type="text" autocomplete="off" placeholder="https://ghe.example.com/api/v3" value={values.github_api_url} />
117+ </div>
118+ </details>
119+ <div class="field">
120+ <label for="concurrency">
121+ Repositories at a time <span class="faint">(organization or user imports)</span>
122+ </label>
123+ <input id="concurrency" name="concurrency" type="number" min="1" max="8" value={values.concurrency || "3"} style="max-width:100px" />
124+ </div>
125+ <button class="btn primary">Start import</button>
126+ </div>
127+ </Form>
128+ </div>
129+);
130+
131+const SelectForm: FC<{ t: string; row: ImportRow; candidates: Candidate[]; filter: string; selected: Set<string> }> = ({ t, row, candidates, filter, selected }) => {
132+ const shown = filter ? candidates.filter((c) => c.name.toLowerCase().includes(filter.toLowerCase())) : candidates;
133+ return (
134+ <div style="max-width:760px;margin:0 auto">
135+ <h1>Choose repositories</h1>
136+ <p class="muted">
137+ {candidates.length} repositories in <b>{row.source}</b>. Tick the ones to import.
138+ </p>
139+ <Form action={`/imports/${row.id}/select`} t={t} class="box">
140+ <div class="body">
141+ <div class="row" style="gap:8px">
142+ <input type="text" name="filter" value={filter} placeholder="Filter by name…" autocomplete="off" class="grow" />
143+ <button class="btn" name="do" value="filter">
144+ Filter
145+ </button>
146+ <button class="btn" name="do" value="all">
147+ Select all{filter ? " shown" : ""}
148+ </button>
149+ <button class="btn" name="do" value="none">
150+ Clear
151+ </button>
152+ </div>
153+ <div class="kstack mt" style="margin-top:10px;max-height:420px;overflow:auto">
154+ {shown.length ? (
155+ shown.map((cand) => (
156+ <KCheckbox
157+ name="repositories"
158+ value={cand.name}
159+ checked={selected.has(cand.name)}
160+ label={
161+ <span>
162+ <b>{cand.name}</b> {cand.private ? <span class="badge">private</span> : null} {cand.fork ? <span class="badge">fork</span> : null} {cand.archived ? <span class="badge amber">archived</span> : null}
163+ {cand.description ? <span class="muted small"> — {cand.description}</span> : null}
164+ </span>
165+ }
166+ />
167+ ))
168+ ) : (
169+ <p class="muted small">No repositories match "{filter}".</p>
170+ )}
171+ </div>
172+ {/* Keep selections that are filtered out of view. */}
173+ {filter ? candidates.filter((c) => selected.has(c.name) && !shown.includes(c)).map((c) => <input type="hidden" name="repositories" value={c.name} />) : null}
174+ <p class="muted small">{selected.size} selected.</p>
175+ <button class="btn primary" name="do" value="start">
176+ Import {selected.size || ""} {selected.size === 1 ? "repository" : "repositories"}
177+ </button>
178+ </div>
179+ </Form>
180+ </div>
181+ );
182+};
183+
184+const Steps: FC<{ row: ImportRow }> = ({ row }) => {
185+ const state = JSON.parse(row.steps || "{}") as Record<string, StepState>;
186+ const list = (row.kind === "bulk"
187+ ? [["organization", "Organization variables and secret names"], ["repositories", "Repositories"], ["finalize", "Finish"]]
188+ : [["repository", "Create the repository"], ["git", "Copy branches, tags and history"], ["labels", "Labels and milestones"], ["issues", "Issues and pull requests"], ["comments", "Comments"], ["reviews", "Reviews"], ["review_comments", "Review comments"], ["protection", "Branch protection"], ["variables", "Variables and secret names"], ["releases", "Releases"], ["finalize", "Finish"]]) as Array<[string, string]>;
189+ return (
190+ <ul class="list">
191+ {list.map(([name, title]) => {
192+ const s = state[name] ?? { status: "pending" };
193+ const st = s.status === "done" ? { conclusion: "success" } : s.status === "failed" ? { conclusion: "failure" } : s.status === "skipped" ? { conclusion: "skipped" } : s.status === "running" ? { status: "in_progress" } : { status: "queued" };
194+ return (
195+ <li>
196+ <Status {...st} />
197+ <div class="grow">
198+ <b>{title}</b>
199+ {s.message ? <div class="meta">{s.message}</div> : null}
200+ </div>
201+ {typeof s.count === "number" && s.count > 0 ? <span class="muted small">{s.count}</span> : null}
202+ </li>
203+ );
204+ })}
205+ </ul>
206+ );
207+};
208+
209+const Summary: FC<{ row: ImportRow; secretsHref?: string }> = ({ row, secretsHref }) => {
210+ const s = JSON.parse(row.summary || "{}") as ImportSummary;
211+ const warnings = JSON.parse(row.warnings || "[]") as string[];
212+ return (
213+ <>
214+ {s.counts ? (
215+ <div class="box mt">
216+ <div class="body">
217+ <h3 class="m0">Imported</h3>
218+ <p class="muted small">
219+ {s.counts.issues} issues, {s.counts.pulls} pull requests, {s.counts.comments} comments, {s.counts.reviews} reviews, {s.counts.labels} labels, {s.counts.milestones} milestones.
220+ {s.git ? ` ${s.git.branches} branches, ${s.git.tags} tags, ${s.git.pullRefs} pull request heads${s.git.lfsObjects ? `, ${s.git.lfsObjects} LFS objects (${fmtBytes(s.git.lfsBytes)})` : ""}.` : ""}
221+ </p>
222+ {s.pullRefs?.missing?.length ? <p class="muted small">{s.pullRefs.missing.length} pull requests had no head commit on GitHub (deleted forks).</p> : null}
223+ </div>
224+ </div>
225+ ) : null}
226+ {s.children ? (
227+ <div class="box mt">
228+ <div class="body">
229+ <h3 class="m0">Repositories</h3>
230+ <p class="muted small">
231+ {s.children.succeeded} of {s.children.total} imported{s.children.failed ? `, ${s.children.failed} failed` : ""}{s.children.cancelled ? `, ${s.children.cancelled} cancelled` : ""}.
232+ </p>
233+ </div>
234+ </div>
235+ ) : null}
236+ {s.secretNames?.length || s.orgSecretNames?.length ? (
237+ <div class="box mt">
238+ <div class="body">
239+ <h3 class="m0">Secret names</h3>
240+ <p class="muted small">GitHub never returns secret values, so only the names came across. An admin sets the values.</p>
241+ {s.secretNames?.length ? <p class="small m0">Repository: {s.secretNames.map((n) => <code style="margin-right:6px">{n}</code>)}</p> : null}
242+ {s.orgSecretNames?.length ? <p class="small">Organization: {s.orgSecretNames.map((n) => <code style="margin-right:6px">{n}</code>)}</p> : null}
243+ {secretsHref ? <a class="btn sm" href={secretsHref}>Set secret values</a> : null}
244+ </div>
245+ </div>
246+ ) : null}
247+ {s.findings?.length ? (
248+ <div class="box mt" style="border-color:var(--amber)">
249+ <div class="body">
250+ <h3 class="m0">Possible secrets in history (rotate these)</h3>
251+ <p class="muted small">Push protection found these in old commits; they were imported with an audited bypass because the history is already public on GitHub.</p>
252+ <ul class="small">
253+ {s.findings.slice(0, 50).map((f) => (
254+ <li>
255+ {f.kind} {f.path ? <span class="muted">in {f.path}</span> : null} <code>{f.blob.slice(0, 10)}</code>
256+ </li>
257+ ))}
258+ </ul>
259+ </div>
260+ </div>
261+ ) : null}
262+ {warnings.length ? (
263+ <div class="box mt">
264+ <div class="body">
265+ <h3 class="m0">Warnings</h3>
266+ <ul class="small">
267+ {warnings.map((w) => (
268+ <li>{w}</li>
269+ ))}
270+ </ul>
271+ </div>
272+ </div>
273+ ) : null}
274+ </>
275+ );
276+};
277+
278+async function loadRow(c: AppContext, svc: ImportService, me: Account): Promise<ImportRow> {
279+ const row = await svc.row(prm(c, "id"));
280+ if (!row || !(await svc.visibleTo(row, me))) throw new HttpError(404, "import not found");
281+ return row;
282+}
283+
284+export function registerImports(web: WebApp): void {
285+ // ---------------------------------------------------------------- start
286+
287+ web.get("/new/import", async (c) => {
288+ const me = requireAccount(c);
289+ const owners = [me.login, ...(await c.get("dir").orgsFor(me.id)).filter((o) => o.role !== "member").map((o) => o.login)];
290+ const values = { owner: c.req.query("owner") ?? me.login, mode: c.req.query("mode") === "owner" ? "owner" : "repo", source: c.req.query("source") ?? "" };
291+ return render(c, { title: "Import from GitHub" }, <StartForm t={csrfToken(c)} owners={owners} values={values} />);
292+ });
293+
294+ post(web, "/new/import", async (c, f) => {
295+ const me = requireAccount(c);
296+ const svc = new ImportService(c.env);
297+ const mode = f.s("mode") === "owner" ? "owner" : "repo";
298+ const options = Object.fromEntries(OPTIONS.map(([k]) => [k, f.b(`opt_${k}`)])) as Record<string, boolean>;
299+ const source = f.s("source").trim();
300+ const body = {
301+ ...(mode === "owner" ? { source_owner: source } : { source }),
302+ owner: f.s("owner"),
303+ name: f.s("name").trim() || undefined,
304+ visibility: f.s("visibility") || "keep",
305+ github_token: f.s("github_token").trim() || undefined,
306+ github_api_url: f.s("github_api_url").trim() || undefined,
307+ concurrency: Number(f.s("concurrency")) || 3,
308+ options: { issues: options.issues, labels: options.labels, branch_protection: options.protection, variables: options.variables, releases: options.releases, lfs: options.lfs, secrets: f.s("secrets") === "stop" ? "stop" : "bypass" },
309+ };
310+ try {
311+ const scopes = (c.get("principal") as { scopes?: Array<"read" | "write" | "admin"> }).scopes ?? ["admin"];
312+ const row = await svc.create(me, scopes, body, { draft: mode === "owner" });
313+ return c.redirect(row.status === "draft" ? `/imports/${row.id}/select` : `/imports/${row.id}`, 303);
314+ } catch (e) {
315+ if (!(e instanceof HttpError) || e.status >= 500) throw e;
316+ const owners = [me.login, ...(await c.get("dir").orgsFor(me.id)).filter((o) => o.role !== "member").map((o) => o.login)];
317+ const values = { ...body, source, mode, ...Object.fromEntries(OPTIONS.map(([k]) => [`opt_${k}`, options[k] ? "1" : "0"])), secrets: body.options.secrets } as unknown as Record<string, string>;
318+ return render(c, { title: "Import from GitHub", status: e.status }, <StartForm t={csrfToken(c)} owners={owners} values={values} error={e.detail} />);
319+ }
320+ });
321+
322+ // ---------------------------------------------------------------- bulk: choose repositories
323+
324+ web.get("/imports/:id/select", async (c) => {
325+ const me = requireAccount(c);
326+ const svc = new ImportService(c.env);
327+ const row = await loadRow(c, svc, me);
328+ if (row.kind !== "bulk" || row.status !== "draft") return c.redirect(`/imports/${row.id}`, 302);
329+ const req = JSON.parse(row.request || "{}") as StoredRequest;
330+ const candidates = req.candidates ?? [];
331+ return render(c, { title: "Choose repositories" }, <SelectForm t={csrfToken(c)} row={row} candidates={candidates} filter="" selected={new Set(candidates.filter((c) => !c.fork && !c.archived).map((c) => c.name))} />);
332+ });
333+
334+ post(web, "/imports/:id/select", async (c, f) => {
335+ const me = requireAccount(c);
336+ const svc = new ImportService(c.env);
337+ const row = await loadRow(c, svc, me);
338+ if (row.kind !== "bulk" || row.status !== "draft") return c.redirect(`/imports/${row.id}`, 303);
339+ const candidates = (JSON.parse(row.request || "{}") as StoredRequest).candidates ?? [];
340+ const selected = new Set(f.all("repositories"));
341+ const action = f.s("do");
342+ if (action === "start") {
343+ try {
344+ await svc.select(row.id, me, [...selected]);
345+ return c.redirect(`/imports/${row.id}`, 303);
346+ } catch (e) {
347+ if (!(e instanceof HttpError) || e.status >= 500) throw e;
348+ flash(c, "error", e.detail);
349+ }
350+ }
351+ const filter = f.s("filter").trim();
352+ if (action === "all") for (const cand of candidates) if (!filter || cand.name.toLowerCase().includes(filter.toLowerCase())) selected.add(cand.name);
353+ if (action === "none") selected.clear();
354+ return render(c, { title: "Choose repositories" }, <SelectForm t={csrfToken(c)} row={row} candidates={candidates} filter={filter} selected={selected} />);
355+ });
356+
357+ // ---------------------------------------------------------------- list and detail
358+
359+ web.get("/imports", async (c) => {
360+ const me = requireAccount(c);
361+ const svc = new ImportService(c.env);
362+ const rows = await svc.list(me.id);
363+ return render(
364+ c,
365+ { title: "Imports" },
366+ <div style="max-width:860px;margin:0 auto">
367+ <div class="row between mb">
368+ <h1 class="m0">Imports</h1>
369+ <a class="btn primary sm" href="/new/import">
370+ Import from GitHub
371+ </a>
372+ </div>
373+ {rows.length ? (
374+ <ul class="list box">
375+ {rows.map((r) => (
376+ <li>
377+ <div class="grow">
378+ <a class="title" href={`/imports/${r.id}`}>
379+ {r.source}
380+ </a>{" "}
381+ {statusBadge(r.status)} <span class="muted small">{r.kind === "bulk" ? "organization/user" : "repository"}</span>
382+ <div class="meta">Started <Time at={r.created_at} /></div>
383+ </div>
384+ </li>
385+ ))}
386+ </ul>
387+ ) : (
388+ <div class="empty">No imports yet. <a href="/new/import">Import a repository from GitHub</a>.</div>
389+ )}
390+ </div>,
391+ );
392+ });
393+
394+ web.get("/imports/:id", async (c) => {
395+ const me = requireAccount(c);
396+ const svc = new ImportService(c.env);
397+ const row = await loadRow(c, svc, me);
398+ if (row.status === "draft" && row.kind === "bulk") return c.redirect(`/imports/${row.id}/select`, 302);
399+ const s = JSON.parse(row.summary || "{}") as ImportSummary;
400+ const children = row.kind === "bulk" ? await svc.children(row.id) : [];
401+ const owner = (await c.get("dir").account(row.target_owner_id))?.login ?? "";
402+ const target = row.repo_id ? `${owner}/${row.target_name}` : null;
403+ const running = row.status === "running" || row.status === "queued";
404+ const canSecrets = !!row.repo_id && (s.secretNames?.length || s.orgSecretNames?.length);
405+ const t = csrfToken(c);
406+ return render(
407+ c,
408+ { title: `Import of ${row.source}`, refresh: running ? 4 : undefined },
409+ <div style="max-width:860px;margin:0 auto">
410+ <div class="row between mb">
411+ <h1 class="m0" style="word-break:break-all">
412+ {row.source}
413+ </h1>
414+ {statusBadge(row.status)}
415+ </div>
416+ <p class="muted">
417+ {row.kind === "bulk" ? "Organization or user import" : "Repository import"} into <b>{owner}</b>
418+ {target ? <> → <a href={`/${target}`}>{target}</a></> : null}. Started <Time at={row.created_at} />.
419+ {row.source.includes("/") ? <> From <a href={`https://${webHostOf(row.api_url)}/${row.source}`} rel="noreferrer noopener">GitHub</a>.</> : null}
420+ </p>
421+ {row.error ? <div class="flash error">{row.error}</div> : null}
422+ {running ? <p class="muted small">This page refreshes automatically.</p> : null}
423+ <div class="box">
424+ <div class="body">
425+ <Steps row={row} />
426+ </div>
427+ </div>
428+ {children.length ? (
429+ <div class="box mt">
430+ <div class="body">
431+ <h3 class="m0">Repositories</h3>
432+ <ul class="list">
433+ {children.map((ch) => (
434+ <li>
435+ <div class="grow">
436+ <a class="title" href={`/imports/${ch.id}`}>
437+ {ch.source}
438+ </a>{" "}
439+ {statusBadge(ch.status)}
440+ {ch.error ? <div class="meta">{ch.error}</div> : null}
441+ </div>
442+ {ch.repo_id ? <a class="btn sm" href={`/${owner}/${ch.target_name}`}>Open</a> : null}
443+ </li>
444+ ))}
445+ </ul>
446+ </div>
447+ </div>
448+ ) : null}
449+ <Summary row={row} secretsHref={canSecrets ? `/imports/${row.id}/secrets` : undefined} />
450+ <div class="row mt" style="gap:8px">
451+ {running ? (
452+ <Form action={`/imports/${row.id}/cancel`} t={t} confirm="Cancel this import?">
453+ <button class="btn">Cancel import</button>
454+ </Form>
455+ ) : null}
456+ {(row.status === "failed" || row.status === "cancelled") && !row.parent_id ? (
457+ <Form action={`/imports/${row.id}/retry`} t={t} class="box">
458+ <div class="body">
459+ <h3 class="m0">Retry this import</h3>
460+ <p class="muted small m0">The GitHub token from the first run was discarded. For a private source (or LFS, variables or protection), give a token again; leave it blank for a public repository.</p>
461+ <div class="field">
462+ <label for="retry_token">GitHub token <span class="faint">(optional)</span></label>
463+ <input id="retry_token" name="github_token" type="password" autocomplete="off" placeholder="ghp_… or a fine-grained token" />
464+ </div>
465+ <button class="btn primary">Retry</button>
466+ </div>
467+ </Form>
468+ ) : null}
469+ </div>
470+ </div>,
471+ );
472+ });
473+
474+ post(web, "/imports/:id/cancel", async (c) => {
475+ const me = requireAccount(c);
476+ const svc = new ImportService(c.env);
477+ const row = await loadRow(c, svc, me);
478+ return act(c, `/imports/${row.id}`, () => svc.cancel(row.id, me).then(() => undefined), "Import cancelled.");
479+ });
480+
481+ post(web, "/imports/:id/retry", async (c, f) => {
482+ const me = requireAccount(c);
483+ const svc = new ImportService(c.env);
484+ const row = await loadRow(c, svc, me);
485+ const token = f.s("github_token").trim() || null;
486+ return act(c, `/imports/${row.id}`, () => svc.retry(row.id, me, token).then(() => undefined), "Import restarted.");
487+ });
488+
489+ // ---------------------------------------------------------------- set secret values afterwards
490+
491+ web.get("/imports/:id/secrets", async (c) => {
492+ const me = requireAccount(c);
493+ const svc = new ImportService(c.env);
494+ const row = await loadRow(c, svc, me);
495+ if (!row.repo_id) throw new HttpError(404, "no repository yet");
496+ const repo = (await c.get("dir").repoById(row.repo_id))!;
497+ const { caller } = await loadRepo(c, repo.owner_login, repo.name, "admin");
498+ if (caller.kind === "agent") throw new HttpError(403, "agents cannot set secrets");
499+ const names = (JSON.parse(row.summary || "{}") as ImportSummary).secretNames ?? [];
500+ const existing = new Set((await loadRepo(c, repo.owner_login, repo.name, "admin").then((l) => l.stub.secretNames())).map((s) => s.name));
501+ return render(
502+ c,
503+ { title: "Set secret values" },
504+ <div style="max-width:720px;margin:0 auto">
505+ <h1>Secret values</h1>
506+ <p class="muted">
507+ GitHub never reveals secret values, so only the names were imported for <a href={repoUrl(repo)}>{repo.owner_login}/{repo.name}</a>. Set the values here; they are encrypted at rest and never shown again.
508+ </p>
509+ {names.length ? (
510+ names.map((name) => (
511+ <Form action={`/${repo.owner_login}/${repo.name}/settings/secrets`} t={csrfToken(c)} class="box mb">
512+ <div class="body row" style="align-items:flex-end;gap:10px">
513+ <div class="field grow">
514+ <label>
515+ {name} {existing.has(name) ? <span class="badge green">set</span> : <span class="badge amber">empty</span>}
516+ </label>
517+ <input type="hidden" name="name" value={name} />
518+ <input name="value" type="password" autocomplete="off" placeholder="Value" required />
519+ </div>
520+ <button class="btn primary">Save</button>
521+ </div>
522+ </Form>
523+ ))
524+ ) : (
525+ <div class="empty">No secret names were imported.</div>
526+ )}
527+ <p class="muted small">
528+ You can manage all secrets under the repository's <a href={`/${repo.owner_login}/${repo.name}/settings/secrets`}>settings</a>.
529+ </p>
530+ </div>,
531+ );
532+ });
533+
534+ // ---------------------------------------------------------------- linked GitHub accounts
535+
536+ const renderLinked = async (c: AppContext, challenge: { nonce: string; host: string; apiUrl: string } | null) => {
537+ const me = requireAccount(c);
538+ const svc = new ImportService(c.env);
539+ const links = await svc.links(me.id);
540+ const t = csrfToken(c);
541+ return render(
542+ c,
543+ { title: "Linked accounts" },
544+ <div class="grid left">
545+ <SettingsNav me={me} tab="linked" />
546+ <div style="max-width:640px">
547+ <h1>Linked GitHub accounts</h1>
548+ <p class="muted">When you import from GitHub, work by a linked GitHub account is attributed to your Diffinery account. Prove you control the account by putting a one-time code in a public gist; no token is needed.</p>
549+ {links.length ? (
550+ <ul class="list box mb">
551+ {links.map((l) => (
552+ <li>
553+ <div class="grow">
554+ <b>{l.login}</b> <span class="muted small">on {l.host}</span>
555+ <div class="meta">Linked <Time at={l.createdAt} /></div>
556+ </div>
557+ <Form action="/settings/linked-accounts/delete" t={t} confirm={`Unlink ${l.login}?`}>
558+ <input type="hidden" name="id" value={l.id} />
559+ <button class="btn sm">Unlink</button>
560+ </Form>
561+ </li>
562+ ))}
563+ </ul>
564+ ) : null}
565+ {svc.oauthEnabled ? (
566+ <Form action="/settings/linked-accounts/github/oauth" t={t} class="box mb">
567+ <div class="body">
568+ <h3 class="m0">Link with GitHub</h3>
569+ <p class="muted small">Sign in to GitHub and authorise read-only access to your profile.</p>
570+ <button class="btn primary">Continue to GitHub</button>
571+ </div>
572+ </Form>
573+ ) : null}
574+ {challenge ? (
575+ <div class="box mb">
576+ <div class="body">
577+ <h3 class="m0">Step 2 · verify your gist</h3>
578+ <p class="muted small">Create a <b>public</b> gist on {challenge.host} whose description or a file contains this code, then enter your GitHub username and verify:</p>
579+ <pre class="code" style="user-select:all">{challenge.nonce}</pre>
580+ <Form action="/settings/linked-accounts/github" t={t}>
581+ <input type="hidden" name="api_url" value={challenge.apiUrl} />
582+ <div class="field">
583+ <label for="llogin">Your GitHub username on {challenge.host}</label>
584+ <input id="llogin" name="login" type="text" autocomplete="off" placeholder="octocat" required />
585+ </div>
586+ <button class="btn primary">Verify and link</button>
587+ </Form>
588+ </div>
589+ </div>
590+ ) : null}
591+ <Form action="/settings/linked-accounts/github/challenge" t={t} class="box">
592+ <div class="body">
593+ <h3 class="m0">Step 1 · get a code</h3>
594+ <div class="field">
595+ <label for="lapi">
596+ GitHub Enterprise API URL <span class="faint">(leave blank for github.com)</span>
597+ </label>
598+ <input id="lapi" name="api_url" type="text" autocomplete="off" placeholder="https://ghe.example.com/api/v3" value={challenge?.apiUrl ?? ""} />
599+ </div>
600+ <button class="btn">Get my code</button>
601+ </div>
602+ </Form>
603+ </div>
604+ </div>,
605+ );
606+ };
607+
608+ web.get("/settings/linked-accounts", async (c) => renderLinked(c, null));
609+
610+ post(web, "/settings/linked-accounts/github/challenge", async (c, f) => {
611+ const me = requireAccount(c);
612+ const svc = new ImportService(c.env);
613+ const ch = await svc.linkChallenge(me, f.s("api_url").trim() || null);
614+ return renderLinked(c, { nonce: ch.nonce, host: ch.host, apiUrl: f.s("api_url").trim() });
615+ });
616+
617+ post(web, "/settings/linked-accounts/github", async (c, f) => {
618+ const me = requireAccount(c);
619+ const svc = new ImportService(c.env);
620+ return act(c, "/settings/linked-accounts", async () => {
621+ await svc.linkByGist(me, f.s("login").trim(), f.s("api_url").trim() || null);
622+ }, "GitHub account linked.");
623+ });
624+
625+ post(web, "/settings/linked-accounts/github/oauth", async (c) => {
626+ const me = requireAccount(c);
627+ const svc = new ImportService(c.env);
628+ return c.redirect(await svc.oauthStart(me));
629+ });
630+
631+ web.get("/settings/linked/github/callback", async (c) => {
632+ const me = requireAccount(c);
633+ const svc = new ImportService(c.env);
634+ return act(c, "/settings/linked-accounts", async () => {
635+ await svc.oauthCallback(me, c.req.query("code") ?? "", c.req.query("state") ?? "");
636+ }, "GitHub account linked.");
637+ });
638+
639+ post(web, "/settings/linked-accounts/delete", async (c, f) => {
640+ const me = requireAccount(c);
641+ const svc = new ImportService(c.env);
642+ return act(c, "/settings/linked-accounts", () => svc.unlink(me, f.s("id")), "Account unlinked.");
643+ });
644+}
modifiedapps/diffinery/src/web/repo-settings.tsx+1−1View file
@@ -398,7 +398,7 @@
398398 async function secretsPage(c: AppContext, l: Loaded) {
399399 const base = repoUrl(l.repo);
400400 const t = csrfToken(c);
401- const [secrets, vars] = await Promise.all([l.stub.secretNames(), l.stub.variables()]);
401+ const [secrets, vars] = await Promise.all([l.stub.secretNames(), l.stub.variables(false)]);
402402 return page(
403403 c,
404404 l,
modifiedapps/diffinery/src/web/routes.tsx+3−0View file
@@ -10,6 +10,7 @@
1010 import { registerCollab } from "./collab.js";
1111 import { registerReleases } from "./releases.js";
1212 import { registerRepoSettings } from "./repo-settings.js";
13+import { registerImports } from "./imports.js";
1314 import { registerSeo } from "./seo.js";
1415 import { errorPage } from "./ui.js";
1516 import { registerKumoAssets } from "./kumo.js";
@@ -39,6 +40,8 @@
3940 registerAuth(web);
4041 registerPackages(web);
4142 registerMailWeb(web);
43+// Before registerAccount: its /:owner catch-all would otherwise swallow /imports, /new/import and /settings/linked-accounts.
44+registerImports(web);
4245 registerAccount(web);
4346 registerRepoSettings(web);
4447 registerActions(web);
modifiedapps/diffinery/src/web/ui.tsx+16−2View file
@@ -114,8 +114,19 @@
114114 return <img class={`av ${size ?? ""} ${kind === "agent" || kind === "system" ? "bot" : ""}`} src={avatarPath(login, px, version)} width={css} height={css} alt="" loading="lazy" decoding="async" />;
115115 };
116116
117-/** Linked actor name with an "agent" badge and the human it acts for. */
118-export const Actor: FC<{ login: string; kind?: string; onBehalfOf?: string | null; avatar?: boolean; version?: number | null }> = ({ login, kind, onBehalfOf, avatar = true, version }) => (
117+/**
118+ * Linked actor name with an "agent" badge and the human it acts for. People imported from
119+ * GitHub without a Diffinery account (kind "github", or logins stored as "github:<login>")
120+ * are shown with a "GitHub" badge and no link.
121+ */
122+export const Actor: FC<{ login: string; kind?: string; onBehalfOf?: string | null; avatar?: boolean; version?: number | null }> = ({ login, kind, onBehalfOf, avatar = true, version }) =>
123+ kind === "github" || login.startsWith("github:") ? (
124+ <span class="who" title="Imported from GitHub; not linked to a Diffinery account">
125+ {avatar ? <Avatar login={login.replace(/^github:/, "")} /> : null}
126+ <span>{login.replace(/^github:/, "")}</span>
127+ <span class="badge">GitHub</span>
128+ </span>
129+ ) : (
119130 <span class="who">
120131 {avatar ? <Avatar login={login} kind={kind} version={version} /> : null}
121132 {kind === "system" ? <span>{login}</span> : <a href={`/${encodeURIComponent(login)}`} style="color:inherit">{login}</a>}
@@ -274,6 +285,8 @@
274285 head?: Child;
275286 /** A page script, run after the site's own (it gets the page's CSP nonce). */
276287 script?: string;
288+ /** Reload the page every this many seconds (progress pages without live updates). */
289+ refresh?: number;
277290 }
278291
279292 export const WATCH_OPTIONS = [
@@ -401,6 +414,7 @@
401414 <meta charset="utf-8" />
402415 <meta name="viewport" content="width=device-width, initial-scale=1" />
403416 <meta name="color-scheme" content={theme ?? "light dark"} />
417+ {opts.refresh ? <meta http-equiv="refresh" content={String(opts.refresh)} /> : null}
404418 <title>{title}</title>
405419 {SeoHead(c, {
406420 title,
modifiedapps/diffinery/test/e2e/e2e.test.mjs+360−0View file
@@ -18,6 +18,7 @@
1818 import { crc32, deflateSync } from "node:zlib";
1919 import { test, before, after } from "node:test";
2020 import assert from "node:assert/strict";
21+import { fakeGitHub, lfsObject } from "./fake-github.mjs";
2122
2223 const APP = resolve(dirname(fileURLToPath(import.meta.url)), "../..");
2324 const AGENT = resolve(APP, "../../containers/runner/dist/agent.mjs");
@@ -1891,3 +1892,362 @@
18911892 assert.deepEqual(await del.json(), { avatar_url: `${base}/api/avatars/${me.login}` });
18921893 assert.match((await avatar(me.login, "?s=64")).type, /^image\/svg\+xml/);
18931894 });
1895+
1896+// ---------------------------------------------------------------- GitHub imports
1897+
1898+/** Build the fake GitHub's repositories with real git: history with a merged pull request, a draft, a fork head, tags, LFS and an old secret. */
1899+function githubFixture(root) {
1900+ const g = (cwd, ...args) => execFileSync("git", args, { cwd, encoding: "utf8", env: { ...process.env, GIT_AUTHOR_NAME: "Octo", GIT_AUTHOR_EMAIL: "octo@example.com", GIT_COMMITTER_NAME: "Octo", GIT_COMMITTER_EMAIL: "octo@example.com" } }).trim();
1901+ const src = join(root, "work-app");
1902+ mkdirSync(join(src, ".github/workflows"), { recursive: true });
1903+ g(root, "init", "-q", "-b", "main", src);
1904+ const asset = lfsObject(`binary asset ${randomBytes(8).toString("hex")}\n`);
1905+ writeFileSync(join(src, "README.md"), "# app\n");
1906+ writeFileSync(join(src, ".gitattributes"), "*.bin filter=lfs diff=lfs merge=lfs -text\n");
1907+ writeFileSync(join(src, "asset.bin"), asset.pointer);
1908+ // Would start a run on every push if imports fired workflows.
1909+ writeFileSync(join(src, ".github/workflows/ci.yml"), "name: CI\non: [push, pull_request]\njobs:\n build:\n runs-on: [self-hosted, linux]\n steps:\n - run: echo hi\n");
1910+ // A credential committed long ago (built here so this file has none).
1911+ writeFileSync(join(src, "settings.env"), `AWS_ACCESS_KEY_ID=${"AKIA"}${"Q3EGRK5V7XJZ2MWN"}\n`);
1912+ g(src, "add", "-A");
1913+ g(src, "-c", "core.hooksPath=/dev/null", "commit", "-q", "-m", "initial");
1914+ g(src, "tag", "v0.1");
1915+ g(src, "rm", "-q", "settings.env");
1916+ g(src, "commit", "-q", "-m", "remove credentials");
1917+ // Enough data that a small push budget needs several chunks.
1918+ for (let i = 0; i < 4; i++) {
1919+ writeFileSync(join(src, `data${i}.bin.txt`), randomBytes(90 * 1024).toString("base64"));
1920+ g(src, "add", "-A");
1921+ g(src, "commit", "-q", "-m", `data ${i}`);
1922+ }
1923+ g(src, "checkout", "-q", "-b", "feature");
1924+ writeFileSync(join(src, "feature.txt"), "feature\n");
1925+ g(src, "add", "-A");
1926+ g(src, "commit", "-q", "-m", "feature");
1927+ const featureSha = g(src, "rev-parse", "HEAD");
1928+ g(src, "checkout", "-q", "main");
1929+ g(src, "merge", "-q", "--no-ff", "-m", "Merge pull request #2 from octo/feature", "feature");
1930+ const mergeSha = g(src, "rev-parse", "HEAD");
1931+ g(src, "tag", "-a", "v1.0", "-m", "Version 1.0");
1932+ g(src, "checkout", "-q", "-b", "draft-work");
1933+ writeFileSync(join(src, "draft.txt"), "wip\n");
1934+ g(src, "add", "-A");
1935+ g(src, "commit", "-q", "-m", "draft");
1936+ const draftSha = g(src, "rev-parse", "HEAD");
1937+ // A fork's commit: only GitHub's refs/pull/6/head points at it.
1938+ g(src, "checkout", "-q", "--detach", "main");
1939+ writeFileSync(join(src, "fork.txt"), "from a fork\n");
1940+ g(src, "add", "-A");
1941+ g(src, "commit", "-q", "-m", "fork change");
1942+ const forkSha = g(src, "rev-parse", "HEAD");
1943+ g(src, "checkout", "-q", "main");
1944+ const bare = join(root, "octo", "app.git");
1945+ mkdirSync(join(root, "octo"), { recursive: true });
1946+ g(root, "clone", "-q", "--bare", src, bare);
1947+ g(bare, "update-ref", "refs/pull/2/head", featureSha);
1948+ g(bare, "update-ref", "refs/pull/2/merge", mergeSha);
1949+ g(bare, "update-ref", "refs/pull/5/head", draftSha);
1950+ g(bare, "update-ref", "refs/pull/6/head", forkSha);
1951+ g(bare, "update-ref", "refs/heads/gh-readonly-queue/main/pr-9-0123456789abcdef", mergeSha);
1952+ // A second repository for the organization import.
1953+ const lib = join(root, "work-lib");
1954+ g(root, "init", "-q", "-b", "trunk", lib);
1955+ writeFileSync(join(lib, "lib.txt"), "lib\n");
1956+ g(lib, "add", "-A");
1957+ g(lib, "commit", "-q", "-m", "lib");
1958+ g(root, "clone", "-q", "--bare", lib, join(root, "octo", "lib.git"));
1959+ return { asset, featureSha, mergeSha, draftSha, forkSha, mainSha: mergeSha };
1960+}
1961+
1962+test("import from GitHub: git data, issues and pull requests with their numbers, no side effects", async () => {
1963+ const me = await ok("GET", "/user");
1964+ const owner = me.login;
1965+ const ghRoot = join(work, "github");
1966+ const fx = githubFixture(ghRoot);
1967+ const GH_TOKEN = `ghfake${randomBytes(12).toString("hex")}`;
1968+ const t = "2024-01-02T03:04:05Z";
1969+ const u = (login, id) => ({ login, id, type: "User" });
1970+ const octoDev = u("octo-dev", 1001);
1971+ const alice = u("alice", 2001);
1972+ const bob = u("bob", 3001);
1973+ let ghBase;
1974+ let linkNonce = ""; // set from the link challenge, then served back in octo-dev's public gist
1975+ const pr = (number, extra) => ({ number, title: `PR ${number}`, body: `Body of #${number}`, state: "open", user: alice, labels: [], assignees: [], milestone: null, locked: false, created_at: t, updated_at: t, closed_at: null, draft: false, merged_at: null, merge_commit_sha: null, requested_reviewers: [], head: { ref: "x", sha: fx.mainSha, label: "octo:x", repo: { full_name: "octo/app" } }, base: { ref: "main", repo: { full_name: "octo/app" } }, ...extra });
1976+ const issue = (number, extra) => ({ number, title: `Issue ${number}`, body: `Body of #${number}`, state: "open", state_reason: null, user: octoDev, labels: [], assignees: [], milestone: null, locked: false, created_at: t, updated_at: t, closed_at: null, ...extra });
1977+ const pulls = [
1978+ pr(2, { title: "Add feature", state: "closed", closed_at: "2024-02-01T00:00:00Z", merged_at: "2024-02-01T00:00:00Z", merge_commit_sha: fx.mergeSha, labels: [{ name: "docs" }], head: { ref: "feature", sha: fx.featureSha, label: "octo:feature", repo: { full_name: "octo/app" } } }),
1979+ pr(5, { title: "Work in progress", draft: true, requested_reviewers: [bob], head: { ref: "draft-work", sha: fx.draftSha, label: "octo:draft-work", repo: { full_name: "octo/app" } } }),
1980+ pr(6, { title: "From a fork", state: "closed", closed_at: "2024-03-01T00:00:00Z", head: { ref: "patch", sha: fx.forkSha, label: "forker:patch", repo: { full_name: "forker/app" } } }),
1981+ ];
1982+ const issues = [
1983+ issue(1, { title: "First bug", labels: [{ name: "bug", color: "d73a4a" }], assignees: [octoDev], milestone: { number: 1 } }),
1984+ ...pulls.map((p) => ({ ...p, pull_request: { url: "x" } })),
1985+ issue(3, { title: "Won't fix", user: alice, state: "closed", state_reason: "not_planned", closed_at: "2024-01-05T00:00:00Z", locked: true }),
1986+ // #4 was deleted on GitHub: the number stays unused.
1987+ issue(7, { title: "Latest", user: { login: "ghost-user", id: 9999 } }),
1988+ ];
1989+ const repoJson = (name, extra) => ({ name, full_name: `octo/${name}`, owner: { login: "octo" }, private: true, visibility: "private", fork: false, archived: false, description: `The ${name}`, homepage: "https://app.example.com", topics: ["demo", "import"], default_branch: "main", ...extra });
1990+ const empty = [];
1991+ const routes = {
1992+ "GET /user": octoDev,
1993+ // Gist-proof account linking: octo-dev's public profile, their gists, and the gist body with the nonce.
1994+ "GET /users/octo-dev": octoDev,
1995+ "GET /users/octo-dev/gists": () => [{ id: "proofgist", public: true, description: "", files: { "proof.md": { raw_url: `${ghBase}/raw/proofgist` } } }],
1996+ "GET /gists/proofgist": () => ({ files: { "proof.md": { content: `linking to diffinery: ${linkNonce}` } } }),
1997+ "GET /users/octo": { login: "octo", type: "Organization" },
1998+ "GET /orgs/octo/repos": () => [repoJson("app"), repoJson("lib", { default_branch: "trunk", topics: [] })].map((r) => ({ ...r, clone_url: `${ghBase}/${r.full_name}.git` })),
1999+ "GET /orgs/octo/actions/variables": { variables: [{ name: "ORG_REGION", value: "eu-west" }] },
2000+ "GET /orgs/octo/actions/secrets": { secrets: [{ name: "ORG_DEPLOY_KEY" }] },
2001+ "GET /repos/octo/app": () => ({ ...repoJson("app"), clone_url: `${ghBase}/octo/app.git`, html_url: `${ghBase}/octo/app` }),
2002+ "GET /repos/octo/app/labels": [{ name: "bug", color: "d73a4a", description: "Something isn't working" }, { name: "docs", color: "0075ca", description: "Documentation" }, { name: "wontfix", color: "ffffff", description: "" }],
2003+ "GET /repos/octo/app/milestones": [{ number: 1, title: "v1", description: "First release", state: "open", due_on: "2030-01-01T00:00:00Z", created_at: t, updated_at: t, closed_at: null, creator: octoDev }],
2004+ "GET /repos/octo/app/pulls": pulls,
2005+ "GET /repos/octo/app/issues": issues,
2006+ "GET /repos/octo/app/issues/comments": () => [
2007+ { id: 11, body: "I can reproduce this", user: alice, created_at: "2024-01-03T00:00:00Z", updated_at: "2024-01-03T00:00:00Z", issue_url: `${ghBase}/api/v3/repos/octo/app/issues/1` },
2008+ { id: 12, body: "Thanks!", user: octoDev, created_at: "2024-01-04T00:00:00Z", updated_at: "2024-01-04T00:00:00Z", issue_url: `${ghBase}/api/v3/repos/octo/app/issues/1` },
2009+ { id: 13, body: "Merging this", user: octoDev, created_at: "2024-01-31T00:00:00Z", updated_at: "2024-01-31T00:00:00Z", issue_url: `${ghBase}/api/v3/repos/octo/app/issues/2` },
2010+ ],
2011+ "GET /repos/octo/app/pulls/2/reviews": [{ id: 501, user: bob, state: "APPROVED", body: "Looks good", commit_id: fx.featureSha, submitted_at: "2024-01-30T00:00:00Z" }],
2012+ "GET /repos/octo/app/pulls/5/reviews": [{ id: 505, user: octoDev, state: "APPROVED", body: "looks fine", commit_id: fx.draftSha, submitted_at: "2024-01-20T00:00:00Z" }],
2013+ "GET /repos/octo/app/pulls/6/reviews": [{ id: 502, user: alice, state: "PENDING", body: "", commit_id: fx.forkSha, submitted_at: null }],
2014+ "GET /repos/octo/app/pulls/2": { ...pulls[0], merged_by: octoDev },
2015+ "GET /repos/octo/app/pulls/comments": () => [
2016+ { id: 601, pull_request_review_id: 501, pull_request_url: `${ghBase}/api/v3/repos/octo/app/pulls/2`, user: bob, body: "Nice file", path: "feature.txt", line: 1, side: "RIGHT", commit_id: fx.featureSha, created_at: "2024-01-30T00:00:00Z", updated_at: "2024-01-30T00:00:00Z" },
2017+ { id: 602, pull_request_review_id: 503, pull_request_url: `${ghBase}/api/v3/repos/octo/app/pulls/2`, in_reply_to_id: 601, user: alice, body: "Agreed", path: "feature.txt", line: 1, side: "RIGHT", commit_id: fx.featureSha, created_at: "2024-01-30T01:00:00Z", updated_at: "2024-01-30T01:00:00Z" },
2018+ ],
2019+ "GET /repos/octo/app/branches": [{ name: "main" }],
2020+ "GET /repos/octo/app/branches/main/protection": { required_status_checks: { contexts: ["build"] }, required_pull_request_reviews: { required_approving_review_count: 1, dismiss_stale_reviews: true }, enforce_admins: { enabled: false }, allow_force_pushes: { enabled: false } },
2021+ "GET /repos/octo/app/actions/variables": { variables: [{ name: "DEPLOY_ENV", value: "staging" }, { name: "REGION", value: "us" }] },
2022+ "GET /repos/octo/app/actions/secrets": { secrets: [{ name: "NPM_TOKEN" }, { name: "SLACK_WEBHOOK" }] },
2023+ "GET /repos/octo/app/releases": [{ id: 1, tag_name: "v1.0.0", name: "1.0.0", body: "first release", draft: false, prerelease: false, created_at: "2024-01-01T00:00:00Z", published_at: "2024-01-01T00:00:00Z", author: { login: "octocat", id: 1 }, assets: [] }],
2024+ "GET /repos/octo/lib": () => ({ ...repoJson("lib", { default_branch: "trunk", topics: [] }), clone_url: `${ghBase}/octo/lib.git` }),
2025+ "GET /repos/octo/lib/labels": empty,
2026+ "GET /repos/octo/lib/milestones": empty,
2027+ "GET /repos/octo/lib/pulls": empty,
2028+ "GET /repos/octo/lib/issues": [issue(1, { title: "Lib issue" })],
2029+ "GET /repos/octo/lib/issues/comments": empty,
2030+ "GET /repos/octo/lib/pulls/comments": empty,
2031+ "GET /repos/octo/lib/branches": empty,
2032+ "GET /repos/octo/lib/actions/variables": { variables: [] },
2033+ "GET /repos/octo/lib/actions/secrets": { secrets: [] },
2034+ "GET /repos/octo/lib/releases": empty,
2035+ };
2036+ const gh = fakeGitHub({ root: ghRoot, token: GH_TOKEN, routes, lfs: new Map([[fx.asset.oid, fx.asset.data]]) });
2037+ ghBase = await gh.listen();
2038+ const apiUrl = `${ghBase}/api/v3`;
2039+
2040+ /** Run the import agent (the runner image's import mode) for an import waiting for one, like the container would. */
2041+ const transfer = async (id) => {
2042+ await until(`import ${id} to wait for an agent`, async () => (await ok("GET", `/diffinery/imports/${id}`)).git?.phase === "waiting", 120_000);
2043+ const { token: pushToken } = await ok("POST", `/diffinery/imports/${id}/claim`);
2044+ const r = await spawnAsync(process.execPath, [AGENT], { DIFFINERY_API_URL: base, DIFFINERY_IMPORT_ID: id, DIFFINERY_IMPORT_TOKEN: pushToken, DIFFINERY_IMPORT_CHUNK_BYTES: String(160 * 1024), DIFFINERY_WORK_ROOT: join(work, `import-${id}`) }, 240_000);
2045+ const out = `${r.stdout}${r.stderr}`;
2046+ assert.ok(!out.includes(GH_TOKEN) && !out.includes(pushToken), "the agent never prints tokens");
2047+ return { ...r, out, pushToken };
2048+ };
2049+ const finished = (id, timeout = 180_000) =>
2050+ until(`import ${id} to finish`, async () => {
2051+ const r = await ok("GET", `/diffinery/imports/${id}`);
2052+ return ["succeeded", "failed", "cancelled"].includes(r.status) ? r : null;
2053+ }, timeout);
2054+
2055+ // ------------------------------------------------------------ people: link a GitHub account by gist proof (no token)
2056+ const challenge = await ok("POST", "/diffinery/user/linked-accounts/github/challenge", { api_url: apiUrl });
2057+ assert.match(challenge.nonce, /^diffinery-link-[0-9a-f]+$/);
2058+ // Before the gist carries the nonce, linking is refused.
2059+ assert.equal((await api("POST", "/diffinery/user/linked-accounts/github", { login: "octo-dev", api_url: apiUrl })).status, 422);
2060+ linkNonce = challenge.nonce; // now octo-dev's public gist contains it
2061+ const link = await ok("POST", "/diffinery/user/linked-accounts/github", { login: "octo-dev", api_url: apiUrl });
2062+ assert.equal(link.login, "octo-dev");
2063+
2064+ // ------------------------------------------------------------ input validation and permissions
2065+ assert.equal((await api("POST", "/diffinery/imports", { source: "octo/app", owner, options: { nope: true }, github_api_url: apiUrl })).status, 422);
2066+ assert.equal((await api("POST", "/diffinery/imports", { source: "octo/app", owner: "nobody-here", github_api_url: apiUrl })).status, 422);
2067+ assert.equal((await api("POST", "/diffinery/imports", { source: "octo/app", owner, github_token: "wrong", github_api_url: apiUrl })).status, 422);
2068+ const bot = (await ok("POST", "/diffinery/agents", { login: "import-bot" })).token.token;
2069+ const botTry = await api("POST", "/diffinery/imports", { source: "octo/app", owner, github_token: GH_TOKEN, github_api_url: apiUrl }, bot);
2070+ assert.equal(botTry.status, 403, "agents cannot import into a person's account");
2071+
2072+ // ------------------------------------------------------------ one repository
2073+ const started = await ok("POST", "/diffinery/imports", { source: "octo/app", owner, name: "imported-app", github_token: GH_TOKEN, github_api_url: apiUrl });
2074+ assert.equal(started.status, "queued");
2075+ assert.ok(!JSON.stringify(started).includes(GH_TOKEN), "the token is never returned");
2076+ // While the Git transfer waits, a webhook is added: nothing the import does may reach it.
2077+ await until("the repository to be created", async () => (await api("GET", `/repos/${owner}/imported-app`)).status === 200, 120_000);
2078+ const hook = await ok("POST", `/repos/${owner}/imported-app/hooks`, { config: { url: "https://hooks.invalid/import-sink" }, events: ["*"] });
2079+ const agent = await transfer(started.id);
2080+ assert.equal(agent.status, 0, agent.out);
2081+ const done = await finished(started.id);
2082+ assert.equal(done.status, "succeeded", JSON.stringify(done, null, 2));
2083+ // No side effects: no workflow runs (ci.yml runs on push and pull_request), no webhook deliveries.
2084+ await sleep(2000);
2085+ assert.equal((await ok("GET", `/repos/${owner}/imported-app/actions/runs`)).total_count, 0);
2086+ const sink = (await ok("GET", `/repos/${owner}/imported-app/hooks`)).find((h) => h.id === hook.id);
2087+ assert.equal(sink.last_response.code, null, "no webhook was delivered");
2088+ assert.equal(sink.last_response.status, "unused");
2089+ assert.ok(done.steps.every((s) => s.status === "done" || s.status === "skipped"), JSON.stringify(done.steps));
2090+ assert.ok(done.summary.git.chunks >= 3, `history went in chunks: ${JSON.stringify(done.summary.git)}`);
2091+ assert.equal(done.summary.git.branches, 3);
2092+ assert.equal(done.summary.git.tags, 2);
2093+ assert.equal(done.summary.git.lfsObjects, 1);
2094+ assert.deepEqual(done.summary.secretNames, ["NPM_TOKEN", "SLACK_WEBHOOK"]);
2095+ assert.ok(done.summary.findings.some((f) => f.kind === "AWS access key"), "bypassed findings are listed");
2096+ assert.ok(done.summary.people.imported.includes("alice") && done.summary.people.linked.includes("octo-dev"));
2097+ // Releases are imported over the REST API (not just their tags).
2098+ assert.equal(done.summary.releases?.imported, 1, JSON.stringify(done.summary.releases));
2099+ const releases = await ok("GET", `/repos/${owner}/imported-app/releases`);
2100+ assert.deepEqual(releases.map((r) => r.tag_name), ["v1.0.0"]);
2101+
2102+ // Repository settings
2103+ const repo = await ok("GET", `/repos/${owner}/imported-app`);
2104+ assert.equal(repo.visibility, "private");
2105+ assert.equal(repo.default_branch, "main");
2106+ assert.equal(repo.description, "The app");
2107+ assert.equal(repo.homepage, "https://app.example.com");
2108+ assert.deepEqual(repo.topics, ["demo", "import"]);
2109+
2110+ // Git: branches, tags and pull request heads; no merge queue branches, merge refs or import refs.
2111+ const refs = Object.fromEntries(
2112+ git(work, ["ls-remote", `${base}/${owner}/imported-app.git`])
2113+ .trim()
2114+ .split("\n")
2115+ .map((l) => l.split("\t").reverse()),
2116+ );
2117+ assert.equal(refs["refs/heads/main"], fx.mainSha);
2118+ assert.equal(refs["refs/heads/feature"], fx.featureSha);
2119+ assert.equal(refs["refs/heads/draft-work"], fx.draftSha);
2120+ assert.ok(refs["refs/tags/v1.0"] && refs["refs/tags/v0.1"]);
2121+ assert.equal(refs["refs/pull/2/head"], fx.featureSha);
2122+ assert.equal(refs["refs/pull/5/head"], fx.draftSha);
2123+ assert.equal(refs["refs/pull/6/head"], fx.forkSha);
2124+ assert.ok(!Object.keys(refs).some((r) => r.startsWith("refs/heads/gh-readonly-queue/") || r.startsWith("refs/diffinery-import/") || r === "refs/pull/2/merge"), Object.keys(refs).join("\n"));
2125+ const copy = join(work, "imported-app");
2126+ git(work, ["clone", "-q", "--mirror", `${base}/${owner}/imported-app.git`, copy], token, { env: { ...process.env, GIT_LFS_SKIP_SMUDGE: "1", GIT_TERMINAL_PROMPT: "0" } });
2127+ git(copy, ["fsck", "--no-progress"]);
2128+ git(copy, ["cat-file", "-e", fx.forkSha]);
2129+ // The LFS object is stored in Diffinery.
2130+ const lfsRes = await fetch(`${base}/${owner}/imported-app.git/info/lfs/objects/batch`, { method: "POST", headers: { authorization: `Bearer ${token}`, "content-type": "application/vnd.git-lfs+json" }, body: JSON.stringify({ operation: "download", objects: [{ oid: fx.asset.oid, size: fx.asset.data.length }] }) });
2131+ const lfsBody = await lfsRes.json();
2132+ assert.ok(lfsBody.objects[0].actions?.download, `${JSON.stringify(lfsBody)}\n${JSON.stringify(done.warnings)}\n${agent.out}\n${devLog.split("\n").filter((l) => /lfs|rror/i.test(l)).slice(-40).join("\n")}`);
2133+ const blob = await fetch(lfsBody.objects[0].actions.download.href, { headers: { authorization: `Bearer ${token}` } });
2134+ assert.equal(Buffer.from(await blob.arrayBuffer()).toString(), fx.asset.data.toString());
2135+ // The push token is revoked.
2136+ assert.equal((await api("GET", "/user", undefined, agent.pushToken)).status, 401);
2137+
2138+ // Issues keep their numbers, people, state and dates.
2139+ const i1 = await ok("GET", `/repos/${owner}/imported-app/issues/1`);
2140+ assert.equal(i1.title, "First bug");
2141+ assert.equal(i1.user.login, owner, "a linked GitHub account maps to its Diffinery account");
2142+ assert.deepEqual(i1.assignees.map((a) => a.login), [owner]);
2143+ assert.deepEqual(i1.labels.map((l) => l.name), ["bug"]);
2144+ assert.equal(i1.milestone.title, "v1");
2145+ assert.equal(i1.created_at, t);
2146+ const i3 = await ok("GET", `/repos/${owner}/imported-app/issues/3`);
2147+ assert.equal(i3.state, "closed");
2148+ assert.equal(i3.state_reason, "not_planned");
2149+ assert.equal(i3.user.login, "github:alice");
2150+ assert.equal((await api("GET", `/repos/${owner}/imported-app/issues/4`)).status, 404, "a deleted GitHub issue leaves its number unused");
2151+ const c1 = await ok("GET", `/repos/${owner}/imported-app/issues/1/comments`);
2152+ assert.deepEqual(c1.map((c) => [c.user.login, c.body, c.created_at]), [["github:alice", "I can reproduce this", "2024-01-03T00:00:00Z"], [owner, "Thanks!", "2024-01-04T00:00:00Z"]]);
2153+ // Pull requests: merged state, who merged, reviews and line comments with replies.
2154+ const p2 = await ok("GET", `/repos/${owner}/imported-app/pulls/2`);
2155+ assert.equal(p2.merged, true);
2156+ assert.equal(p2.merged_by.login, owner);
2157+ assert.equal(p2.merge_commit_sha, fx.mergeSha);
2158+ assert.equal(p2.head.ref, "feature");
2159+ assert.equal(p2.head.sha, fx.featureSha);
2160+ const reviews = await ok("GET", `/repos/${owner}/imported-app/pulls/2/reviews`);
2161+ assert.deepEqual(reviews.map((r) => [r.user.login, r.state, r.body]), [["github:bob", "APPROVED", "Looks good"]]);
2162+ const lineComments = await ok("GET", `/repos/${owner}/imported-app/pulls/2/comments`);
2163+ assert.equal(lineComments.length, 2);
2164+ assert.equal(lineComments[0].path, "feature.txt");
2165+ assert.equal(lineComments[1].in_reply_to_id, lineComments[0].id);
2166+ const p5 = await ok("GET", `/repos/${owner}/imported-app/pulls/5`);
2167+ assert.equal(p5.draft, true);
2168+ assert.equal(p5.state, "open");
2169+ assert.deepEqual(p5.requested_reviewers.map((r) => r.login), ["github:bob"]);
2170+ // M1: an imported review is history, not a vote — even when its author is a linked Diffinery account.
2171+ const r5 = await ok("GET", `/repos/${owner}/imported-app/pulls/5/reviews`);
2172+ assert.deepEqual(r5.map((r) => [r.user.login, r.state]), [[owner, "APPROVED"]], "the imported approval is kept as history");
2173+ assert.equal(p5.approvals, 0, "an imported approval does not count toward the required approvals");
2174+ assert.equal(p5.required_approvals, 1);
2175+ const p6 = await ok("GET", `/repos/${owner}/imported-app/pulls/6`);
2176+ assert.equal(p6.head.ref, "forker:patch");
2177+ assert.equal(p6.head.sha, fx.forkSha);
2178+ assert.equal(p6.merged, false);
2179+ // New issues continue after the highest imported number.
2180+ assert.equal((await ok("POST", `/repos/${owner}/imported-app/issues`, { title: "After import" })).number, 8);
2181+ // Labels, milestones, protection, variables.
2182+ const labels = await ok("GET", `/repos/${owner}/imported-app/labels`);
2183+ {
2184+ const bug = labels.find((l) => l.name === "bug");
2185+ assert.deepEqual([bug.name, bug.color, bug.description], ["bug", "d73a4a", "Something isn't working"]);
2186+ }
2187+ const milestones = await ok("GET", `/repos/${owner}/imported-app/milestones?state=all`);
2188+ assert.equal(milestones.length, 1);
2189+ assert.equal(milestones[0].due_on, "2030-01-01T00:00:00Z");
2190+ assert.equal(milestones[0].open_issues, 1);
2191+ const protection = await ok("GET", `/repos/${owner}/imported-app/branches/main/protection`);
2192+ assert.match(JSON.stringify(protection), /CI \/ build/);
2193+ const vars = await ok("GET", `/repos/${owner}/imported-app/actions/variables`);
2194+ assert.deepEqual(vars.variables.map((v) => [v.name, v.value]).sort(), [["DEPLOY_ENV", "staging"], ["REGION", "us"]]);
2195+ assert.ok((await ok("GET", "/diffinery/imports")).some((x) => x.id === started.id));
2196+
2197+ // ------------------------------------------------------------ secrets policy "stop" fails with a clear message
2198+ const strict = await ok("POST", "/diffinery/imports", { source: "octo/app", owner, name: "strict-app", github_token: GH_TOKEN, github_api_url: apiUrl, options: { secrets: "stop", issues: false } });
2199+ const strictAgent = await transfer(strict.id);
2200+ assert.equal(strictAgent.status, 1, strictAgent.out);
2201+ assert.match(strictAgent.out, /secret/i);
2202+ const strictDone = await finished(strict.id);
2203+ assert.equal(strictDone.status, "failed");
2204+ assert.match(strictDone.error, /secrets/);
2205+
2206+ // ------------------------------------------------------------ a whole organization, into an organization
2207+ await ok("POST", "/user/orgs", { login: "acme-imports" });
2208+ const bulk = await ok("POST", "/diffinery/imports", { source_owner: "octo", owner: "acme-imports", github_token: GH_TOKEN, github_api_url: apiUrl, concurrency: 2, options: { lfs: false } });
2209+ assert.equal(bulk.kind, "organization");
2210+ assert.deepEqual(bulk.children.map((ch) => ch.source).sort(), ["octo/app", "octo/lib"]);
2211+ for (const ch of bulk.children) assert.equal((await transfer(ch.id)).status, 0);
2212+ const bulkDone = await finished(bulk.id, 300_000);
2213+ assert.equal(bulkDone.status, "succeeded", JSON.stringify(bulkDone, null, 2));
2214+ assert.deepEqual(bulkDone.summary.orgSecretNames, ["ORG_DEPLOY_KEY"]);
2215+ const orgVars = await ok("GET", "/orgs/acme-imports/actions/variables");
2216+ assert.deepEqual(orgVars.variables.map((v) => [v.name, v.value]), [["ORG_REGION", "eu-west"]]);
2217+ assert.equal((await ok("GET", "/repos/acme-imports/lib")).default_branch, "trunk");
2218+ assert.equal((await ok("GET", "/repos/acme-imports/lib/issues/1")).title, "Lib issue");
2219+ assert.equal((await ok("GET", "/repos/acme-imports/app/pulls/2")).merged, true);
2220+
2221+ // ------------------------------------------------------------ an agent's organization import cannot touch variables (H1)
2222+ const agentBulk = await api("POST", "/diffinery/imports", { source_owner: "octo", owner: "acme-imports", github_token: GH_TOKEN, github_api_url: apiUrl, concurrency: 1, options: { lfs: false } }, bot);
2223+ assert.equal(agentBulk.status, 201, JSON.stringify(agentBulk.json));
2224+ const agentBulkDone = await finished(agentBulk.json.id, 180_000);
2225+ const orgStep = agentBulkDone.steps.find((s) => s.name === "organization");
2226+ assert.equal(orgStep.status, "skipped", JSON.stringify(agentBulkDone.steps));
2227+ assert.match(orgStep.message, /agents cannot manage variables/);
2228+ // The organization's variable is untouched by the agent import.
2229+ assert.deepEqual((await ok("GET", "/orgs/acme-imports/actions/variables")).variables.map((v) => v.name), ["ORG_REGION"]);
2230+
2231+ // ------------------------------------------------------------ a retry onto a repository adopted since is refused (H2)
2232+ await ok("POST", `/repos/${owner}/strict-app/issues`, { title: "adopted after the failed import" });
2233+ await ok("POST", `/diffinery/imports/${strict.id}/retry`, { github_token: GH_TOKEN });
2234+ const reDone = await finished(strict.id);
2235+ assert.equal(reDone.status, "failed");
2236+ assert.match(reDone.error, /holds content|start a new import/);
2237+
2238+ // ------------------------------------------------------------ a stale (revoked) import push token cannot push in import mode (M4)
2239+ const stalePush = gitFails(copy, ["push", `${base}/${owner}/imported-app.git`, "--push-option=diffinery-import", "refs/heads/main:refs/heads/stale-attempt"], agent.pushToken);
2240+ assert.match(stalePush, /401|403|denied|forbidden|authentication|fatal/i, stalePush);
2241+
2242+ // ------------------------------------------------------------ the web UI (signed in via DEV_AUTH)
2243+ for (const path of ["/new/import", "/imports", `/imports/${started.id}`, `/imports/${started.id}/secrets`, `/imports/${bulk.id}`, "/settings/linked-accounts", `/${owner}/imported-app/issues/1`, `/${owner}/imported-app/pull/2`]) {
2244+ const res = await fetch(`${base}${path}`);
2245+ const html = await res.text();
2246+ assert.equal(res.status, 200, `${path} -> ${res.status}\n${html.slice(0, 500)}`);
2247+ assert.ok(!/<select|type="checkbox"|type="radio"/.test(html.replace(/<template[\s\S]*?<\/template>/g, "")) || path.startsWith(`/${owner}`), `${path} renders Kumo controls, not native ones`);
2248+ }
2249+ assert.match(await (await fetch(`${base}/imports/${started.id}`)).text(), /NPM_TOKEN/);
2250+ assert.match(await (await fetch(`${base}/${owner}/imported-app/issues/1`)).text(), /v1/);
2251+ assert.match(await (await fetch(`${base}/${owner}/imported-app/issues/3`)).text(), /alice/);
2252+ await gh.close();
2253+});
modifiedapps/diffinery/test/e2e/entry.ts+2−0View file
@@ -7,5 +7,7 @@
77 export { RepoDO } from "../../src/repo/RepoDO.js";
88 export { RunnerContainer, SandboxContainer } from "../../src/runner/containers.js";
99 export { RunWorkflow } from "../../src/runner/workflow.js";
10+export { ImportWorkflow } from "../../src/imports/workflow.js";
11+export { ImportContainer } from "../../src/imports/container.js";
1012
1113 export default { fetch: app.fetch, queue, scheduled } satisfies ExportedHandler<Env, DiffineryEvent | MailJob>;
addedapps/diffinery/test/e2e/fake-github.mjs+146−0View file
@@ -1,0 +1,146 @@
1+// A stand-in for GitHub (or GitHub Enterprise) for the import end-to-end test:
2+// the REST API under /api/v3 with Link pagination, real Git over smart HTTP
3+// (`git http-backend`) and a Git LFS batch endpoint. Everything requires the
4+// token it was created with, like a private repository would.
5+import { spawn } from "node:child_process";
6+import { createServer } from "node:http";
7+import { createHash } from "node:crypto";
8+
9+/**
10+ * @param {object} o
11+ * @param {string} o.root directory holding <owner>/<name>.git bare repositories
12+ * @param {string} o.token the token every request must carry
13+ * @param {Record<string, unknown>} o.routes "GET /repos/o/r/labels" → JSON (arrays are paginated)
14+ * @param {Map<string, Buffer>} o.lfs LFS objects by oid
15+ */
16+export function fakeGitHub({ root, token, routes, lfs }) {
17+ const requests = [];
18+ let base;
19+ const authorized = (req) => {
20+ const h = req.headers.authorization ?? "";
21+ if (h === `Bearer ${token}` || h === `token ${token}`) return true;
22+ const basic = /^Basic (.+)$/.exec(h);
23+ return !!basic && Buffer.from(basic[1], "base64").toString().split(":").slice(1).join(":") === token;
24+ };
25+ const json = (res, status, body, headers = {}) => {
26+ res.writeHead(status, { "content-type": "application/json", ...headers });
27+ res.end(JSON.stringify(body));
28+ };
29+ const server = createServer(async (req, res) => {
30+ const url = new URL(req.url, base);
31+ const chunks = [];
32+ for await (const c of req) chunks.push(c);
33+ const body = Buffer.concat(chunks);
34+ requests.push({ method: req.method, path: url.pathname, query: url.search });
35+ // Public, unauthenticated GitHub endpoints: a user profile and their public gists (used by gist-proof account linking).
36+ const publicGet = req.method === "GET" && /^\/api\/v3\/(users\/[^/]+(?:\/gists)?|gists\/[^/]+)$/.test(url.pathname);
37+ if (!publicGet && !authorized(req)) return json(res, 401, { message: "Bad credentials" }, { "www-authenticate": 'Basic realm="GitHub"' });
38+
39+ // ---------------------------------------------------------------- Git LFS
40+ const lfsBatch = /^\/([^/]+)\/([^/]+?)(?:\.git)?\/info\/lfs\/objects\/batch$/.exec(url.pathname);
41+ if (lfsBatch && req.method === "POST") {
42+ const q = JSON.parse(body.toString());
43+ const objects = q.objects.map((o) =>
44+ lfs.has(o.oid)
45+ ? { oid: o.oid, size: o.size, authenticated: true, actions: { download: { href: `${base}/lfs-objects/${o.oid}`, header: { Authorization: req.headers.authorization } } } }
46+ : { oid: o.oid, size: o.size, error: { code: 404, message: "not found" } },
47+ );
48+ res.writeHead(200, { "content-type": "application/vnd.git-lfs+json" });
49+ return res.end(JSON.stringify({ transfer: "basic", objects }));
50+ }
51+ const lfsGet = /^\/lfs-objects\/([0-9a-f]{64})$/.exec(url.pathname);
52+ if (lfsGet) {
53+ const data = lfs.get(lfsGet[1]);
54+ if (!data) return json(res, 404, { message: "Not Found" });
55+ res.writeHead(200, { "content-type": "application/octet-stream", "content-length": data.length });
56+ return res.end(data);
57+ }
58+
59+ // ---------------------------------------------------------------- Git smart HTTP
60+ if (/^\/[^/]+\/[^/]+\.git\/(info\/refs|git-upload-pack)$/.test(url.pathname)) {
61+ return cgi(req, res, url, body, root);
62+ }
63+
64+ // ---------------------------------------------------------------- REST
65+ if (!url.pathname.startsWith("/api/v3/")) return json(res, 404, { message: "Not Found" });
66+ const path = url.pathname.slice("/api/v3".length);
67+ const key = `${req.method} ${path}`;
68+ const query = Object.fromEntries(url.searchParams);
69+ const route = routes[key];
70+ if (route === undefined) return json(res, 404, { message: "Not Found" });
71+ const value = typeof route === "function" ? route(query) : route;
72+ // Lists are paginated two items at a time to exercise Link headers.
73+ const list = Array.isArray(value) ? value : value && typeof value === "object" && Array.isArray(value.variables ?? value.secrets) ? (value.variables ?? value.secrets) : null;
74+ if (!list) return json(res, 200, value);
75+ const per = Math.min(Number(query.per_page ?? 30), 2);
76+ const page = Number(query.page ?? 1);
77+ const items = list.slice((page - 1) * per, page * per);
78+ const headers = {};
79+ if (page * per < list.length) {
80+ const next = new URL(url);
81+ next.searchParams.set("page", String(page + 1));
82+ headers.link = `<${next}>; rel="next"`;
83+ }
84+ const out = Array.isArray(value) ? items : { total_count: list.length, [value.variables ? "variables" : "secrets"]: items };
85+ return json(res, 200, out, headers);
86+ });
87+ return {
88+ requests,
89+ async listen() {
90+ await new Promise((r) => server.listen(0, "127.0.0.1", r));
91+ server.unref();
92+ base = `http://127.0.0.1:${server.address().port}`;
93+ return base;
94+ },
95+ close: () => new Promise((r) => server.close(r)),
96+ };
97+}
98+
99+function cgi(req, res, url, body, root) {
100+ const env = {
101+ ...process.env,
102+ GIT_PROJECT_ROOT: root,
103+ GIT_HTTP_EXPORT_ALL: "1",
104+ PATH_INFO: url.pathname,
105+ REQUEST_METHOD: req.method,
106+ QUERY_STRING: url.search.slice(1),
107+ CONTENT_TYPE: req.headers["content-type"] ?? "",
108+ CONTENT_LENGTH: String(body.length),
109+ HTTP_CONTENT_ENCODING: req.headers["content-encoding"] ?? "",
110+ GIT_PROTOCOL: req.headers["git-protocol"] ?? "",
111+ REMOTE_USER: "importer",
112+ REMOTE_ADDR: "127.0.0.1",
113+ };
114+ const child = spawn("git", ["http-backend"], { env });
115+ const out = [];
116+ child.stdout.on("data", (d) => out.push(d));
117+ child.stderr.on("data", () => {});
118+ // git http-backend may close stdin before reading the whole body (e.g. GET info/refs); ignore the broken pipe.
119+ child.stdin.on("error", () => {});
120+ child.on("error", () => {});
121+ child.stdin.end(body);
122+ child.on("close", () => {
123+ if (res.writableEnded) return;
124+ const buf = Buffer.concat(out);
125+ const sep = buf.indexOf("\r\n\r\n");
126+ const head = buf.subarray(0, sep).toString();
127+ const headers = {};
128+ let status = 200;
129+ for (const line of head.split("\r\n")) {
130+ const i = line.indexOf(":");
131+ const k = line.slice(0, i).trim();
132+ const v = line.slice(i + 1).trim();
133+ if (k.toLowerCase() === "status") status = Number(v.split(" ")[0]);
134+ else headers[k] = v;
135+ }
136+ res.writeHead(status, headers);
137+ res.end(buf.subarray(sep + 4));
138+ });
139+}
140+
141+/** An LFS pointer file and the object it points to. */
142+export function lfsObject(content) {
143+ const data = Buffer.from(content);
144+ const oid = createHash("sha256").update(data).digest("hex");
145+ return { oid, data, pointer: `version https://git-lfs.github.com/spec/v1\noid sha256:${oid}\nsize ${data.length}\n` };
146+}
addedapps/diffinery/test/unit/imports.test.ts+222−0View file
@@ -1,0 +1,222 @@
1+import { DatabaseSync } from "node:sqlite";
2+import { describe, expect, it } from "vitest";
3+import { checkPublicUrl, isPrivateAddress, normalizeApiUrl, parseImportRequest, parseOptions, parseRepoSource, safeAssetName, visibilityFor } from "../../src/imports/options.js";
4+import { People, mapIssue, mapPull, mapReview, type GhIssue, type GhPull, type Link } from "../../src/imports/mapping.js";
5+import { RepoImports } from "../../src/repo/imports.js";
6+import { REPO_SCHEMA, migrateRepoSchema } from "../../src/repo/schema.js";
7+
8+// ---------------------------------------------------------------- request and option validation
9+
10+describe("import request validation", () => {
11+ const dev = { dev: false };
12+
13+ it("accepts a single repository and defaults the name and visibility", () => {
14+ const r = parseImportRequest({ source: "octo/App", owner: "me" }, dev);
15+ expect(r.mode).toBe("repo");
16+ expect(r.source).toBe("octo/App");
17+ expect(r.name).toBe("App");
18+ expect(r.visibility).toBe("keep");
19+ expect(r.apiUrl).toBe("https://api.github.com");
20+ });
21+
22+ it("derives the Enterprise API base from a repository URL", () => {
23+ const r = parseImportRequest({ source: "https://ghe.example.com/octo/app", owner: "me" }, dev);
24+ expect(r.apiUrl).toBe("https://ghe.example.com/api/v3");
25+ });
26+
27+ it("maps GitHub internal visibility to private", () => {
28+ expect(parseImportRequest({ source: "o/r", owner: "me", visibility: "internal" }, dev).visibility).toBe("private");
29+ });
30+
31+ it("rejects an unknown visibility and a bad name", () => {
32+ expect(() => parseImportRequest({ source: "o/r", owner: "me", visibility: "secret" }, dev)).toThrow(/visibility/);
33+ expect(() => parseImportRequest({ source: "o/r", owner: "me", name: "../etc" }, dev)).toThrow(/not a valid repository name/);
34+ });
35+
36+ it("requires exactly one of source and source_owner", () => {
37+ expect(() => parseImportRequest({ owner: "me" }, dev)).toThrow(/either source/);
38+ expect(() => parseImportRequest({ source: "o/r", source_owner: "o", owner: "me" }, dev)).toThrow(/either source/);
39+ });
40+
41+ it("validates a bulk selection and its size", () => {
42+ const r = parseImportRequest({ source_owner: "octo", owner: "me", repositories: ["a", "a", "b"] }, dev);
43+ expect(r.mode).toBe("bulk");
44+ expect(r.repositories).toEqual(["a", "b"]);
45+ expect(() => parseImportRequest({ source_owner: "octo", owner: "me", repositories: [] }, dev)).toThrow(/omit it/);
46+ expect(() => parseImportRequest({ source_owner: "octo", owner: "me", concurrency: 99 }, dev)).toThrow(/concurrency/);
47+ });
48+
49+ it("rejects a token that looks malformed", () => {
50+ expect(() => parseImportRequest({ source: "o/r", owner: "me", github_token: "has spaces" }, dev)).toThrow(/token/);
51+ });
52+
53+ it("reads known options and rejects unknown ones", () => {
54+ const o = parseOptions({ issues: false, branch_protection: false, secrets: "stop" });
55+ expect(o.issues).toBe(false);
56+ expect(o.protection).toBe(false);
57+ expect(o.secrets).toBe("stop");
58+ expect(() => parseOptions({ nope: true })).toThrow(/unknown option/);
59+ expect(() => parseOptions({ secrets: "maybe" })).toThrow(/bypass/);
60+ });
61+});
62+
63+describe("source and visibility helpers", () => {
64+ it("parses owner/name, URLs and SSH remotes", () => {
65+ expect(parseRepoSource("octo/app")).toMatchObject({ owner: "octo", name: "app", host: null });
66+ expect(parseRepoSource("https://github.com/octo/app.git")).toMatchObject({ owner: "octo", name: "app", host: null });
67+ expect(parseRepoSource("git@ghe.example.com:octo/app.git")).toMatchObject({ owner: "octo", name: "app", host: "ghe.example.com" });
68+ });
69+
70+ it("maps every GitHub visibility to public or private", () => {
71+ expect(visibilityFor("keep", { visibility: "public" })).toBe("public");
72+ expect(visibilityFor("keep", { visibility: "internal" })).toBe("private");
73+ expect(visibilityFor("keep", { private: true })).toBe("private");
74+ expect(visibilityFor("keep", { private: false })).toBe("public");
75+ expect(visibilityFor("public", { private: true })).toBe("public");
76+ expect(visibilityFor("private", { visibility: "public" })).toBe("private");
77+ });
78+
79+ it("normalises GitHub.com and Enterprise API bases", () => {
80+ expect(normalizeApiUrl("", false)).toBe("https://api.github.com");
81+ expect(normalizeApiUrl("https://github.com", false)).toBe("https://api.github.com");
82+ expect(normalizeApiUrl("https://ghe.example.com", false)).toBe("https://ghe.example.com/api/v3");
83+ });
84+});
85+
86+describe("SSRF guard", () => {
87+ it("allows public https and refuses private, loopback and bare hosts", () => {
88+ expect(() => checkPublicUrl("https://api.github.com", false)).not.toThrow();
89+ expect(() => checkPublicUrl("http://api.github.com", false)).toThrow(/https/);
90+ expect(() => checkPublicUrl("https://localhost", false)).toThrow(/public host/);
91+ expect(() => checkPublicUrl("https://169.254.169.254", false)).toThrow(/public host/);
92+ expect(() => checkPublicUrl("https://10.0.0.1", false)).toThrow(/public host/);
93+ expect(() => checkPublicUrl("https://user:pw@example.com", false)).toThrow(/credentials/);
94+ // Loopback and plain http are allowed only in development (for test doubles).
95+ expect(() => checkPublicUrl("http://127.0.0.1:8787", true)).not.toThrow();
96+ });
97+
98+ it("recognises private address ranges", () => {
99+ for (const h of ["127.0.0.1", "10.1.2.3", "192.168.0.1", "172.16.0.1", "169.254.0.1", "100.64.0.1", "::1", "fd00::1"]) expect(isPrivateAddress(h)).toBe(true);
100+ for (const h of ["8.8.8.8", "140.82.112.3", "2606:2800::1"]) expect(isPrivateAddress(h)).toBe(false);
101+ });
102+});
103+
104+// ---------------------------------------------------------------- people mapping (verified links only)
105+
106+describe("people mapping", () => {
107+ const links: Link[] = [{ externalId: "42", externalLogin: "octocat", accountId: "acc_1", login: "octo" }];
108+
109+ it("maps a GitHub user to a Diffinery account by numeric id, surviving a rename", () => {
110+ const people = new People(links);
111+ // Linked by id even though the login on the object differs (GitHub logins can be renamed).
112+ const p = people.person({ login: "renamed-login", id: 42 });
113+ expect(p).toEqual({ id: "acc_1", login: "octo", kind: "user" });
114+ expect(people.login({ login: "renamed-login", id: 42 })).toBe("octo");
115+ });
116+
117+ it("leaves unlinked users as imported GitHub users and never invents accounts", () => {
118+ const people = new People(links);
119+ const p = people.person({ login: "stranger", id: 7 });
120+ expect(p).toEqual({ id: "github:7", login: "stranger", kind: "github" });
121+ expect(people.login({ login: "stranger", id: 7 })).toBe("github:stranger");
122+ expect(people.person(null)).toMatchObject({ kind: "github", login: "ghost" });
123+ });
124+});
125+
126+// ---------------------------------------------------------------- number-preserving mapping
127+
128+const issue = (over: Partial<GhIssue>): GhIssue => ({ number: 1, title: "t", body: "b", state: "open", user: { login: "a", id: 1 }, created_at: "2024-01-01T00:00:00Z", updated_at: "2024-01-01T00:00:00Z", ...over });
129+
130+describe("issue and pull-request mapping keeps numbers", () => {
131+ const people = new People([]);
132+
133+ it("keeps the GitHub issue number and maps the author", () => {
134+ const m = mapIssue(issue({ number: 57, state: "closed", state_reason: "completed" }), people);
135+ expect(m.number).toBe(57);
136+ expect(m.kind).toBe("issue");
137+ expect(m.state).toBe("closed");
138+ expect(m.author.kind).toBe("github");
139+ });
140+
141+ it("maps a merged pull request with its number and merge metadata", () => {
142+ const pull: GhPull = {
143+ ...issue({ number: 12, state: "closed" }),
144+ merged_at: "2024-02-02T00:00:00Z",
145+ merged_by: { login: "mergebot", id: 9 },
146+ merge_commit_sha: "abc123",
147+ head: { ref: "feature", sha: "deadbeef", repo: { full_name: "octo/app" } },
148+ base: { ref: "main", sha: "cafe", repo: { full_name: "octo/app" } },
149+ };
150+ const m = mapPull(pull, "octo/app", people);
151+ expect(m.number).toBe(12);
152+ expect(m.kind).toBe("pull");
153+ expect(m.pull?.merged).toBe(true);
154+ expect(m.pull?.headRef).toBe("feature");
155+ expect(m.pull?.baseRef).toBe("main");
156+ });
157+
158+ it("maps a review, skipping pending ones", () => {
159+ expect(mapReview({ id: 1, user: { login: "r", id: 2 }, state: "PENDING" }, 5, people)).toBeNull();
160+ const m = mapReview({ id: 1, user: { login: "r", id: 2 }, state: "APPROVED", submitted_at: "2024-03-03T00:00:00Z" }, 5, people);
161+ expect(m?.number).toBe(5);
162+ expect(m?.state).toBe("APPROVED");
163+ });
164+});
165+
166+// ---------------------------------------------------------------- safe server-side names
167+
168+describe("safeAssetName", () => {
169+ it("keeps a plain filename and strips any path", () => {
170+ expect(safeAssetName("app-linux-amd64.tar.gz")).toBe("app-linux-amd64.tar.gz");
171+ expect(safeAssetName("dir/../evil/app.bin")).toBe("app.bin");
172+ expect(safeAssetName("a\\b\\c.zip")).toBe("c.zip");
173+ });
174+
175+ it("rejects empty, dot and .git names, and control characters", () => {
176+ for (const n of ["", ".", "..", ".git", ".GIT", "git~1", "\0", "..."]) expect(safeAssetName(n)).toBeNull();
177+ expect(safeAssetName(null)).toBeNull();
178+ });
179+});
180+
181+// ---------------------------------------------------------------- numbers preserved in storage
182+
183+/** A minimal SqlStorage over node:sqlite, enough for RepoImports' inserts and reads. */
184+function sqlStorage(): SqlStorage {
185+ const db = new DatabaseSync(":memory:");
186+ db.exec(REPO_SCHEMA);
187+ const sql = {
188+ exec(query: string, ...bindings: unknown[]) {
189+ const stmt = db.prepare(query);
190+ if (/^\s*select/i.test(query)) {
191+ const rows = stmt.all(...(bindings as never[]));
192+ return { toArray: () => rows, one: () => rows[0] } as unknown as SqlStorageCursor<Record<string, SqlStorageValue>>;
193+ }
194+ stmt.run(...(bindings as never[]));
195+ return { toArray: () => [], one: () => undefined } as unknown as SqlStorageCursor<Record<string, SqlStorageValue>>;
196+ },
197+ } as unknown as SqlStorage;
198+ // Apply the ALTER-added columns (e.g. the `imported` provenance columns) just as RepoDO does.
199+ migrateRepoSchema(sql);
200+ return sql;
201+}
202+
203+describe("imports keep original issue and pull-request numbers", () => {
204+ it("inserts with the GitHub numbers, raises the counter and reports counts", () => {
205+ const imp = new RepoImports(sqlStorage());
206+ const people = new People([]);
207+ const rows = [
208+ mapIssue(issue({ number: 1, title: "first" }), people),
209+ mapIssue(issue({ number: 5, title: "a later issue" }), people),
210+ mapPull({ ...issue({ number: 2, title: "a pull" }), head: { ref: "f", sha: "s", repo: { full_name: "o/r" } }, base: { ref: "main", repo: { full_name: "o/r" } } }, "o/r", people),
211+ mapIssue(issue({ number: 42, title: "high number" }), people),
212+ ];
213+ expect(imp.issues(rows, "imp-1")).toBe(4);
214+
215+ const counts = imp.counts();
216+ expect(counts.issues).toBe(3); // issue rows that are not pulls
217+ expect(counts.pulls).toBe(1);
218+ expect(counts.maxNumber).toBe(42);
219+ // The gap at 3 and 4 is preserved: no renumbering.
220+ expect(imp.pullNumbers(0, 10).map((p) => p.number)).toEqual([2]);
221+ });
222+});
addedapps/diffinery/test/unit/stubs/cloudflare-workflows.ts+7−0View file
@@ -1,0 +1,7 @@
1+/** Stand-in for the Workers runtime's `cloudflare:workflows` module, for unit tests that load the router. */
2+export class NonRetryableError extends Error {
3+ constructor(message: string, name?: string) {
4+ super(message);
5+ this.name = name ?? "NonRetryableError";
6+ }
7+}
modifiedapps/diffinery/vitest.config.ts+2−0View file
@@ -1,9 +1,11 @@
1+import { fileURLToPath } from "node:url";
12 import { defineConfig } from "vitest/config";
23
34 // The path is base64-encoded so the id does not look like a node_modules file, which vitest would hand to Node unprocessed.
45 const PREFIX = "virtual:wasm-module/";
56
67 export default defineConfig({
8+ resolve: { alias: { "cloudflare:workflows": fileURLToPath(new URL("./test/unit/stubs/cloudflare-workflows.ts", import.meta.url)) } },
79 plugins: [
810 {
911 // `import mod from "x.wasm?module"` gives a compiled WebAssembly.Module, like wrangler's CompiledWasm rule in the Worker.
modifiedapps/diffinery/wrangler.jsonc+22−7View file
@@ -44,14 +44,23 @@
4444 "bindings": [
4545 { "name": "REPO", "class_name": "RepoDO" },
4646 { "name": "RUNNER", "class_name": "RunnerContainer" },
47- { "name": "SANDBOX", "class_name": "SandboxContainer" }
47+ { "name": "SANDBOX", "class_name": "SandboxContainer" },
48+ { "name": "IMPORTER", "class_name": "ImportContainer" }
4849 ]
4950 },
50- "migrations": [{ "tag": "v1", "new_sqlite_classes": ["RepoDO", "RunnerContainer", "SandboxContainer"] }],
51- "workflows": [{ "name": "diffinery-run", "binding": "RUN_WORKFLOW", "class_name": "RunWorkflow" }],
51+ "migrations": [
52+ { "tag": "v1", "new_sqlite_classes": ["RepoDO", "RunnerContainer", "SandboxContainer"] },
53+ { "tag": "v2", "new_sqlite_classes": ["ImportContainer"] }
54+ ],
55+ "workflows": [
56+ { "name": "diffinery-run", "binding": "RUN_WORKFLOW", "class_name": "RunWorkflow" },
57+ { "name": "diffinery-import", "binding": "IMPORT_WORKFLOW", "class_name": "ImportWorkflow" }
58+ ],
5259 "containers": [
5360 { "class_name": "RunnerContainer", "image": "../../containers/runner/Dockerfile", "image_build_context": "../../containers/runner", "instance_type": "standard-2", "max_instances": 100, "rollout_active_grace_period": 3600 },
54- { "class_name": "SandboxContainer", "image": "../../containers/runner/Dockerfile", "image_build_context": "../../containers/runner", "instance_type": "standard-1", "max_instances": 50, "rollout_active_grace_period": 3600 }
61+ { "class_name": "SandboxContainer", "image": "../../containers/runner/Dockerfile", "image_build_context": "../../containers/runner", "instance_type": "standard-1", "max_instances": 50, "rollout_active_grace_period": 3600 },
62+ // GitHub imports: a mirror clone of a large repository needs disk and memory.
63+ { "class_name": "ImportContainer", "image": "../../containers/runner/Dockerfile", "image_build_context": "../../containers/runner", "instance_type": "standard-4", "max_instances": 20, "rollout_active_grace_period": 3600 }
5564 ],
5665 "triggers": { "crons": ["*/5 * * * *"] },
5766
@@ -66,7 +75,9 @@
6675 "BOOTSTRAP_ADMINS": "dev@diffinery.local",
6776 "ACCESS_TEAM_DOMAIN": "",
6877 "ACCESS_AUD": "",
69- "DEV_AUTH": "dev@diffinery.local"
78+ "DEV_AUTH": "dev@diffinery.local",
79+ // Containers do not run under wrangler dev: an import agent started by hand (or by the e2e test) claims Git transfers.
80+ "IMPORT_GIT_AGENT": "external"
7081 },
7182 "d1_databases": [{ "binding": "DB", "database_name": "diffinery-dev", "database_id": "00000000-0000-0000-0000-000000000000", "migrations_dir": "migrations" }],
7283 "r2_buckets": [{ "binding": "BUCKET", "bucket_name": "diffinery-data-dev" }],
@@ -86,10 +97,14 @@
8697 "bindings": [
8798 { "name": "REPO", "class_name": "RepoDO" },
8899 { "name": "RUNNER", "class_name": "RunnerContainer" },
89- { "name": "SANDBOX", "class_name": "SandboxContainer" }
100+ { "name": "SANDBOX", "class_name": "SandboxContainer" },
101+ { "name": "IMPORTER", "class_name": "ImportContainer" }
90102 ]
91103 },
92- "workflows": [{ "name": "diffinery-run-dev", "binding": "RUN_WORKFLOW", "class_name": "RunWorkflow" }],
104+ "workflows": [
105+ { "name": "diffinery-run-dev", "binding": "RUN_WORKFLOW", "class_name": "RunWorkflow" },
106+ { "name": "diffinery-import-dev", "binding": "IMPORT_WORKFLOW", "class_name": "ImportWorkflow" }
107+ ],
93108 "triggers": { "crons": [] },
94109 "containers": []
95110 }
addedcontainers/runner/src/import-plan.ts+112−0View file
@@ -1,0 +1,112 @@
1+/**
2+ * Pure planning helpers for the import mode (see importer.ts): how to cut a
3+ * repository's history into pushes that each stay under a size budget, and how
4+ * to read what the server said about a push.
5+ */
6+
7+/**
8+ * Binary search for the furthest point of a first-parent chain whose push stays
9+ * within `budget`. `sizeAt(i)` is the size of everything reachable from the i-th
10+ * commit that the server does not have yet, so it grows with `i`. Returns null
11+ * when even the first commit is over budget.
12+ */
13+export async function furthestWithin(n: number, sizeAt: (i: number) => Promise<number>, budget: number): Promise<{ index: number; size: number } | null> {
14+ if (n <= 0) return null;
15+ const last = await sizeAt(n - 1);
16+ if (last <= budget) return { index: n - 1, size: last };
17+ let lo = 0;
18+ let hi = n - 2;
19+ let best: { index: number; size: number } | null = null;
20+ while (lo <= hi) {
21+ const mid = (lo + hi) >> 1;
22+ const size = await sizeAt(mid);
23+ if (size <= budget) {
24+ best = { index: mid, size };
25+ lo = mid + 1;
26+ } else hi = mid - 1;
27+ }
28+ return best;
29+}
30+
31+/** Greedy batches of at most `budget` bytes, keeping order; an item larger than the budget gets a batch of its own. */
32+export function batchBySize<T extends { size: number }>(items: T[], budget: number): T[][] {
33+ const out: T[][] = [];
34+ let cur: T[] = [];
35+ let used = 0;
36+ for (const it of items) {
37+ if (cur.length && used + it.size > budget) {
38+ out.push(cur);
39+ cur = [];
40+ used = 0;
41+ }
42+ cur.push(it);
43+ used += it.size;
44+ }
45+ if (cur.length) out.push(cur);
46+ return out;
47+}
48+
49+/** Push protection findings from `remote:` lines, e.g. "remote: AWS access key in blob 0123456789". */
50+export function parseSecretFindings(output: string): Array<{ kind: string; blob: string }> {
51+ const out: Array<{ kind: string; blob: string }> = [];
52+ const seen = new Set<string>();
53+ for (const m of output.matchAll(/^remote:\s+(.+?) in blob ([0-9a-f]{7,40})\s*$/gm)) {
54+ const key = `${m[1]}:${m[2]}`;
55+ if (seen.has(key)) continue;
56+ seen.add(key);
57+ out.push({ kind: m[1]!.trim(), blob: m[2]! });
58+ }
59+ return out;
60+}
61+
62+/** The push was refused or cut off because the request was too big. */
63+export function isSizeRejection(output: string): boolean {
64+ return /exceeds \d+ ?MB|split it into smaller pushes|HTTP 413|Request Entity Too Large|RPC failed|remote end hung up|HTTP 50[023]|unexpected disconnect|pack exceeds/i.test(output);
65+}
66+
67+/** The push was refused by push protection. */
68+export function isSecretRejection(output: string): boolean {
69+ return /secret detected|Push blocked: possible secrets/i.test(output);
70+}
71+
72+export type RefKind = "branch" | "tag" | "pull";
73+
74+/** Which refs an import copies: branches, tags and pull request heads (not `refs/pull/N/merge`, merge queue branches, notes or others). */
75+export function importRefKind(ref: string): RefKind | null {
76+ if (ref.startsWith("refs/heads/gh-readonly-queue/")) return null;
77+ if (ref.startsWith("refs/heads/")) return "branch";
78+ if (ref.startsWith("refs/tags/")) return "tag";
79+ if (/^refs\/pull\/\d+\/head$/.test(ref)) return "pull";
80+ return null;
81+}
82+
83+/** Push order: the default branch first, then other branches, tags and pull request heads (by number). */
84+export function sortRefsForImport(refs: string[], defaultBranch: string): string[] {
85+ const rank = (r: string) => (r === `refs/heads/${defaultBranch}` ? 0 : r.startsWith("refs/heads/") ? 1 : r.startsWith("refs/tags/") ? 2 : 3);
86+ const pr = (r: string) => Number(/^refs\/pull\/(\d+)\//.exec(r)?.[1] ?? 0);
87+ return [...refs].sort((a, b) => rank(a) - rank(b) || pr(a) - pr(b) || (a < b ? -1 : a > b ? 1 : 0));
88+}
89+
90+/**
91+ * Import-only namespace on the Diffinery side. The server owns `refs/pull/*`, so
92+ * pull request heads land under `pull/N` here and the server creates
93+ * `refs/pull/N/head` from the imported pull request; chunks use `tmp/`.
94+ */
95+export const IMPORT_REF_PREFIX = "refs/diffinery-import/";
96+
97+/** Where a source ref is pushed: branches and tags as they are, `refs/pull/N/head` to `refs/diffinery-import/pull/N`. */
98+export function importDestination(ref: string): string {
99+ const m = /^refs\/pull\/(\d+)\/head$/.exec(ref);
100+ return m ? `${IMPORT_REF_PREFIX}pull/${m[1]}` : ref;
101+}
102+
103+/** `git push --porcelain` result lines: "<flag>\t<src>:<dst>\t<summary>". */
104+export function parsePorcelain(output: string): Array<{ ref: string; ok: boolean; reason: string }> {
105+ const out: Array<{ ref: string; ok: boolean; reason: string }> = [];
106+ for (const line of output.split("\n")) {
107+ const m = /^([ +\-*=!])\t([^\t]*):([^\t]+)\t(.*)$/.exec(line);
108+ if (!m) continue;
109+ out.push({ ref: m[3]!, ok: m[1] !== "!", reason: m[4]!.replace(/^\[[^\]]*\]\s*/, "").replace(/^\((.*)\)$/, "$1") || m[4]! });
110+ }
111+ return out;
112+}
addedcontainers/runner/src/importer.ts+473−0View file
@@ -1,0 +1,473 @@
1+/**
2+ * Import mode: copy a repository's Git data from GitHub into Diffinery.
3+ *
4+ * Workers can't run git and a push request is capped (Diffinery refuses pushes
5+ * over 100 MB, and a repository's Durable Object has 128 MB of memory), so this
6+ * runs in a container: `git clone --mirror` from GitHub, then push history in
7+ * chunks that stay well under the cap (first-parent walks to temporary refs,
8+ * merged side branches first, and object batches for single huge commits), then
9+ * every branch and tag, pull request heads to `refs/diffinery-import/pull/N`
10+ * (the server creates `refs/pull/N/head` from the imported pull request and
11+ * then drops these), then Git LFS objects.
12+ *
13+ * Tokens never appear in argv or logs: git reads them from GIT_CONFIG_* environment
14+ * entries scoped to one host each. Pushes carry `-o diffinery-import`, which the
15+ * server honours only for this import's own token, so they trigger no workflows,
16+ * webhooks or notifications.
17+ */
18+import { mkdir, readdir, rm, stat } from "node:fs/promises";
19+import path from "node:path";
20+import type { ImportGitResult, ImportProgress, ImportSecretFinding, ImportSpec } from "@diffinery/shared";
21+import { RunnerApi } from "./api.js";
22+import { IMPORT_REF_PREFIX, batchBySize, furthestWithin, importDestination, importRefKind, isSecretRejection, isSizeRejection, parsePorcelain, parseSecretFindings, sortRefsForImport } from "./import-plan.js";
23+import { exec } from "./util.js";
24+
25+export interface ImportAgentOptions {
26+ apiUrl: string;
27+ importId: string;
28+ token: string;
29+ workRoot: string;
30+ /** Overrides the spec's push size budget (tests use a small one to exercise chunking). */
31+ chunkBytes?: number;
32+ log?: (line: string) => void;
33+}
34+
35+/** Chunk refs; only these are deleted at the end (pull request heads stay until the server has created refs/pull/N/head). */
36+const TEMP_PREFIX = `${IMPORT_REF_PREFIX}tmp/`;
37+const MIN_BUDGET = 64 * 1024;
38+const REF_BATCH = 200;
39+
40+export class ImportError extends Error {}
41+
42+const errMsg = (e: unknown) => (e instanceof Error ? e.message : String(e));
43+
44+/** `http.<origin>/.extraHeader` entry: the credential goes to that host only. */
45+function authEntry(url: string, user: string, password: string): [string, string] {
46+ return [`http.${new URL(url).origin}/.extraHeader`, `Authorization: Basic ${Buffer.from(`${user}:${password}`).toString("base64")}`];
47+}
48+
49+function withConfig(env: NodeJS.ProcessEnv, entries: Array<[string, string]>): NodeJS.ProcessEnv {
50+ const out = { ...env };
51+ let n = Number(out.GIT_CONFIG_COUNT ?? 0);
52+ if (!Number.isInteger(n) || n < 0) n = 0;
53+ for (const [k, v] of entries) {
54+ out[`GIT_CONFIG_KEY_${n}`] = k;
55+ out[`GIT_CONFIG_VALUE_${n}`] = v;
56+ n++;
57+ }
58+ out.GIT_CONFIG_COUNT = String(n);
59+ return out;
60+}
61+
62+/** Removes anything that looks like a credential from text we report. */
63+function redact(s: string, secrets: string[]): string {
64+ let out = s;
65+ for (const x of secrets) if (x) out = out.split(x).join("***");
66+ return out.replace(/(Authorization:\s*\w+\s+)\S+/gi, "$1***");
67+}
68+
69+class GitRunner {
70+ constructor(
71+ readonly dir: string,
72+ private readonly env: NodeJS.ProcessEnv,
73+ private readonly secrets: string[],
74+ ) {}
75+
76+ async run(args: string[], o: { input?: string; allowFail?: boolean; cwd?: string; maxCapture?: number } = {}): Promise<{ code: number; stdout: string; stderr: string }> {
77+ const r = await exec("git", args, { cwd: o.cwd ?? this.dir, env: this.env, input: o.input, capture: true, maxCapture: o.maxCapture ?? 256 * 1024 * 1024 });
78+ const res = { code: r.exitCode, stdout: r.stdout, stderr: redact(r.stderr, this.secrets) };
79+ if (r.exitCode !== 0 && !o.allowFail) throw new ImportError(`git ${args[0]} failed: ${res.stderr.trim().split("\n").slice(-3).join(" ").slice(0, 500)}`);
80+ return res;
81+ }
82+
83+ async lines(args: string[], input?: string): Promise<string[]> {
84+ return (await this.run(args, { input })).stdout.split("\n").filter(Boolean);
85+ }
86+}
87+
88+/** Fetch the spec, transfer everything, and report the result. Returns the result it reported. */
89+export async function runImportAgent(o: ImportAgentOptions): Promise<ImportGitResult> {
90+ const log = o.log ?? ((l: string) => process.stdout.write(`[diffinery-import] ${l}\n`));
91+ const api = new RunnerApi(`${o.apiUrl.replace(/\/$/, "")}/api/v3/diffinery/imports/${encodeURIComponent(o.importId)}/agent`, o.token);
92+ const spec = (await (await api.request("/spec", { retries: 8 })).json()) as ImportSpec;
93+ let lastProgress = 0;
94+ const progress = async (p: ImportProgress, force = false) => {
95+ log(p.message);
96+ if (!force && Date.now() - lastProgress < 2000) return;
97+ lastProgress = Date.now();
98+ await api.request("/progress", { method: "POST", json: p, retries: 2 }).catch((e) => log(`progress report failed: ${errMsg(e)}`));
99+ };
100+ let result: ImportGitResult;
101+ try {
102+ result = await transfer(spec, { workRoot: o.workRoot, chunkBytes: o.chunkBytes, progress, log });
103+ } catch (e) {
104+ result = emptyResult();
105+ result.ok = false;
106+ result.error = redact(errMsg(e), [spec.githubToken ?? "", spec.pushToken]);
107+ if (e instanceof SecretsFound) result.secretFindings = e.findings;
108+ }
109+ await api.request("/complete", { method: "POST", json: result, retries: 10 });
110+ return result;
111+}
112+
113+class SecretsFound extends ImportError {
114+ constructor(
115+ message: string,
116+ readonly findings: ImportSecretFinding[],
117+ ) {
118+ super(message);
119+ }
120+}
121+
122+function emptyResult(): ImportGitResult {
123+ return { ok: true, branches: 0, tags: 0, pullRefs: 0, chunks: 0, bytes: 0, lfsObjects: 0, lfsBytes: 0, defaultBranch: null, secretFindings: [], warnings: [] };
124+}
125+
126+export interface TransferOptions {
127+ workRoot: string;
128+ chunkBytes?: number;
129+ progress: (p: ImportProgress, force?: boolean) => Promise<void>;
130+ log: (line: string) => void;
131+}
132+
133+/** The whole Git transfer for one spec. Exported for tests. */
134+export async function transfer(spec: ImportSpec, o: TransferOptions): Promise<ImportGitResult> {
135+ const result = emptyResult();
136+ const secrets = [spec.githubToken ?? "", spec.pushToken];
137+ const root = path.join(o.workRoot, `import-${spec.importId.replace(/[^\w-]/g, "")}`);
138+ await rm(root, { recursive: true, force: true });
139+ await mkdir(root, { recursive: true });
140+ const mirror = path.join(root, "mirror.git");
141+ // The clone URL must be https (a loopback host or a file:// path is allowed for local test
142+ // doubles), so the token never travels in the clear and never reaches a non-GitHub host. In
143+ // production the orchestrator always supplies GitHub's https clone URL.
144+ const localDouble = spec.source.cloneUrl.startsWith("file://");
145+ const cloneHost = localDouble ? "" : new URL(spec.source.cloneUrl).hostname.toLowerCase();
146+ const loopback = cloneHost === "localhost" || cloneHost === "127.0.0.1" || cloneHost === "::1";
147+ if (!spec.source.cloneUrl.startsWith("https://") && !loopback && !localDouble) throw new Error("the clone URL must use https");
148+ // Pin the LFS endpoint to each operation's own host so a repository's .lfsconfig cannot redirect
149+ // LFS traffic (and a token) to another host: the source host for the fetch, the target host for
150+ // the push. A command-line `-c lfs.url=` overrides .lfsconfig, so we set it per command rather
151+ // than globally (a global lfs.url would also misdirect the push to Diffinery back at the source).
152+ const sourceLfsUrl = `${spec.source.cloneUrl.replace(/\.git$/, "")}.git/info/lfs`;
153+ const targetLfsUrl = `${spec.target.pushUrl.replace(/\.git$/, "")}.git/info/lfs`;
154+ const entries: Array<[string, string]> = [
155+ ["credential.helper", ""],
156+ ["protocol.version", "2"],
157+ // Never follow an HTTP redirect: a redirect could carry the credential to another host.
158+ ["http.followRedirects", "false"],
159+ ["lfs.locksverify", "false"],
160+ ["lfs.concurrenttransfers", "4"],
161+ ["core.askPass", "true"],
162+ ];
163+ if (spec.githubToken) entries.push(authEntry(spec.source.cloneUrl, "x-access-token", spec.githubToken));
164+ entries.push(authEntry(spec.target.pushUrl, "x-token", spec.pushToken));
165+ const env = withConfig(
166+ {
167+ ...process.env,
168+ GIT_TERMINAL_PROMPT: "0",
169+ GIT_LFS_SKIP_SMUDGE: "1",
170+ GIT_AUTHOR_NAME: "Diffinery import",
171+ GIT_AUTHOR_EMAIL: "import@diffinery.invalid",
172+ GIT_COMMITTER_NAME: "Diffinery import",
173+ GIT_COMMITTER_EMAIL: "import@diffinery.invalid",
174+ },
175+ entries,
176+ );
177+ const git = new GitRunner(mirror, env, secrets);
178+ try {
179+ // ------------------------------------------------------------ clone
180+ await o.progress({ phase: "clone", message: `Cloning ${spec.source.fullName}` }, true);
181+ await git.run(["clone", "--mirror", "--quiet", spec.source.cloneUrl, mirror], { cwd: root });
182+ const refLines = await git.lines(["for-each-ref", "--format=%(objectname) %(objecttype) %(*objectname) %(*objecttype) %(refname)"]);
183+ const refs = new Map<string, { oid: string; commit: string | null }>();
184+ for (const line of refLines) {
185+ const [oid, type, peeled, peeledType, ...rest] = line.split(" ");
186+ const name = rest.join(" ");
187+ if (!importRefKind(name)) continue;
188+ const commit = type === "commit" ? oid! : peeledType === "commit" ? peeled! : null;
189+ refs.set(name, { oid: oid!, commit });
190+ }
191+ const ordered = sortRefsForImport([...refs.keys()], spec.source.defaultBranch);
192+ if (!ordered.length) {
193+ await o.progress({ phase: "done", message: "The GitHub repository is empty" }, true);
194+ return result;
195+ }
196+
197+ // ------------------------------------------------------------ LFS objects from GitHub
198+ if (spec.lfs) {
199+ await o.progress({ phase: "lfs-fetch", message: "Fetching Git LFS objects" }, true);
200+ const f = await git.run(["-c", `lfs.url=${sourceLfsUrl}`, "lfs", "fetch", "--all", "origin"], { allowFail: true });
201+ if (f.code !== 0) result.warnings.push(`Some Git LFS objects could not be fetched from GitHub: ${lastLine(f.stderr)}`);
202+ }
203+
204+ // ------------------------------------------------------------ history, in chunks
205+ const pushOpts = ["-o", "diffinery-import", ...(spec.secrets === "bypass" ? ["-o", `allow-secrets=${spec.bypassReason}`] : [])];
206+ const pusher = new ChunkPusher(git, spec, pushOpts, o, result);
207+ await pusher.loadRemoteState();
208+ for (const name of ordered) {
209+ const ref = refs.get(name)!;
210+ if (ref.commit) await pusher.ensurePushed(ref.commit, name);
211+ }
212+
213+ // ------------------------------------------------------------ the refs themselves
214+ await o.progress({ phase: "refs", message: `Creating ${ordered.length} branches, tags and pull request refs`, chunks: result.chunks, bytes: result.bytes }, true);
215+ for (let i = 0; i < ordered.length; i += REF_BATCH) {
216+ const batch = ordered.slice(i, i + REF_BATCH);
217+ const out = await pusher.push(batch.map((r) => `+${r}:${importDestination(r)}`));
218+ const status = new Map(parsePorcelain(out.stdout).map((s) => [s.ref, s]));
219+ for (const r of batch) {
220+ const s = status.get(importDestination(r));
221+ if (out.code !== 0 && !s) throw new ImportError(`pushing refs failed: ${lastLine(out.stderr)}`);
222+ if (s && !s.ok) {
223+ result.warnings.push(`${r} was not imported: ${s.reason}`);
224+ continue;
225+ }
226+ const kind = importRefKind(r);
227+ if (kind === "branch") result.branches++;
228+ else if (kind === "tag") result.tags++;
229+ else if (kind === "pull") result.pullRefs++;
230+ }
231+ }
232+ if (refs.has(`refs/heads/${spec.source.defaultBranch}`)) result.defaultBranch = spec.source.defaultBranch;
233+
234+ // ------------------------------------------------------------ LFS objects to Diffinery
235+ if (spec.lfs) {
236+ const { count, bytes } = await lfsStore(mirror);
237+ if (count > 0) {
238+ await o.progress({ phase: "lfs-push", message: `Uploading ${count} Git LFS objects` }, true);
239+ const p = await git.run(["-c", `lfs.url=${targetLfsUrl}`, "lfs", "push", "--all", spec.target.pushUrl], { allowFail: true });
240+ if (p.code !== 0) result.warnings.push(`Some Git LFS objects could not be uploaded: ${lastLine(p.stderr)}`);
241+ result.lfsObjects = count;
242+ result.lfsBytes = bytes;
243+ }
244+ }
245+
246+ // ------------------------------------------------------------ cleanup
247+ await o.progress({ phase: "cleanup", message: "Removing temporary refs" }, true);
248+ await pusher.deleteTempRefs();
249+ result.secretFindings = await pusher.findingsWithPaths();
250+ await o.progress({ phase: "done", message: `Imported ${result.branches} branches, ${result.tags} tags and ${result.pullRefs} pull request refs`, chunks: result.chunks, bytes: result.bytes }, true);
251+ return result;
252+ } finally {
253+ await rm(root, { recursive: true, force: true }).catch(() => {});
254+ }
255+}
256+
257+const lastLine = (s: string) =>
258+ s
259+ .trim()
260+ .split("\n")
261+ .filter((l) => l.trim())
262+ .slice(-1)[0]
263+ ?.slice(0, 300) ?? "";
264+
265+/** Count and size of the LFS objects in a bare repository's local store. */
266+async function lfsStore(gitDir: string): Promise<{ count: number; bytes: number }> {
267+ let count = 0;
268+ let bytes = 0;
269+ const walk = async (dir: string): Promise<void> => {
270+ let entries;
271+ try {
272+ entries = await readdir(dir, { withFileTypes: true });
273+ } catch {
274+ return;
275+ }
276+ for (const e of entries) {
277+ const p = path.join(dir, e.name);
278+ if (e.isDirectory()) await walk(p);
279+ else if (/^[0-9a-f]{64}$/.test(e.name)) {
280+ count++;
281+ bytes += (await stat(p)).size;
282+ }
283+ }
284+ };
285+ await walk(path.join(gitDir, "lfs", "objects"));
286+ return { count, bytes };
287+}
288+
289+/**
290+ * Pushes history so that every request stays under the budget. The server keeps
291+ * what it has under temporary refs, so later pushes (and a restarted import) only
292+ * send what is missing.
293+ */
294+class ChunkPusher {
295+ private haves: string[] = [];
296+ private haveSet = new Set<string>();
297+ private budget: number;
298+ private temp = 0;
299+ private tempRefs: string[] = [];
300+ private findings = new Map<string, ImportSecretFinding>();
301+
302+ constructor(
303+ private readonly git: GitRunner,
304+ private readonly spec: ImportSpec,
305+ private readonly pushOpts: string[],
306+ private readonly o: TransferOptions,
307+ private readonly result: ImportGitResult,
308+ ) {
309+ this.budget = Math.max(MIN_BUDGET, o.chunkBytes ?? spec.chunkBytes);
310+ }
311+
312+ /** Objects the server already has (a resumed import) count as pushed. */
313+ async loadRemoteState(): Promise<void> {
314+ const r = await this.git.run(["ls-remote", this.spec.target.pushUrl], { allowFail: true });
315+ if (r.code !== 0) throw new ImportError(`cannot reach the Diffinery repository: ${lastLine(r.stderr)}`);
316+ const oids = [...new Set(r.stdout.split("\n").map((l) => l.split("\t")[0]!).filter((x) => /^[0-9a-f]{40}$/.test(x)))];
317+ if (!oids.length) return;
318+ const present = await this.git.run(["cat-file", "--batch-check=%(objectname) %(objecttype)"], { input: `${oids.join("\n")}\n` });
319+ for (const line of present.stdout.split("\n")) {
320+ const [oid, type] = line.split(" ");
321+ if (type === "commit") this.addHave(oid!);
322+ }
323+ for (const line of r.stdout.split("\n")) {
324+ const ref = line.split("\t")[1];
325+ if (ref?.startsWith(TEMP_PREFIX)) this.tempRefs.push(ref);
326+ }
327+ this.temp = this.tempRefs.length;
328+ }
329+
330+ private addHave(oid: string): void {
331+ if (this.haveSet.has(oid)) return;
332+ this.haveSet.add(oid);
333+ this.haves.push(oid);
334+ }
335+
336+ private notHaves(): string {
337+ return this.haves.map((h) => `^${h}\n`).join("");
338+ }
339+
340+ /** Bytes on disk of everything reachable from `tip` that the server doesn't have. */
341+ async sizeOf(tip: string): Promise<number> {
342+ const r = await this.git.run(["rev-list", "--objects", "--disk-usage", "--stdin"], { input: `${tip}\n${this.notHaves()}` });
343+ return Number(r.stdout.trim()) || 0;
344+ }
345+
346+ /** Make sure the server has `tip` and everything it needs. */
347+ async ensurePushed(tip: string, label: string): Promise<void> {
348+ if (this.haveSet.has(tip)) return;
349+ let chain = await this.git.lines(["rev-list", "--first-parent", "--reverse", "--stdin"], `${tip}\n${this.notHaves()}`);
350+ while (chain.length) {
351+ const sizes = new Map<number, number>();
352+ const best = await furthestWithin(
353+ chain.length,
354+ async (i) => {
355+ const s = await this.sizeOf(chain[i]!);
356+ sizes.set(i, s);
357+ return s;
358+ },
359+ this.budget,
360+ );
361+ if (best) {
362+ if (!(await this.pushTemp(chain[best.index]!, best.size, label))) continue;
363+ chain = chain.slice(best.index + 1);
364+ continue;
365+ }
366+ // One first-parent step is over budget: push the side branches it merges first,
367+ // then, if it still is, its new objects in batches.
368+ const commit = chain[0]!;
369+ const parents = (await this.git.lines(["rev-list", "--parents", "-n", "1", commit]))[0]!.split(" ").slice(2);
370+ for (const p of parents) await this.ensurePushed(p, label);
371+ if ((await this.sizeOf(commit)) > this.budget) await this.pushObjectBatches(commit, label);
372+ if (!(await this.pushTemp(commit, await this.sizeOf(commit), label))) continue;
373+ chain = chain.slice(1);
374+ }
375+ }
376+
377+ /** Upload the blobs `commit` adds through throwaway commits of flat trees, so the commit itself stays small. */
378+ private async pushObjectBatches(commit: string, label: string): Promise<void> {
379+ const listed = await this.git.lines(["rev-list", "--objects", "--stdin"], `${commit}\n${this.notHaves()}`);
380+ const oids = listed.map((l) => l.split(" ")[0]!);
381+ const check = await this.git.lines(["cat-file", "--batch-check=%(objectname) %(objecttype) %(objectsize:disk)"], `${oids.join("\n")}\n`);
382+ const blobs = check
383+ .map((l) => l.split(" "))
384+ .filter((p) => p[1] === "blob")
385+ .map((p) => ({ oid: p[0]!, size: Number(p[2]) }));
386+ const batches = batchBySize(blobs, this.budget);
387+ for (const [i, batch] of batches.entries()) {
388+ const tree = (await this.git.run(["mktree"], { input: batch.map((b, j) => `100644 blob ${b.oid}\tobject-${j}\n`).join("") })).stdout.trim();
389+ const carrier = (await this.git.run(["commit-tree", tree, "-m", `Diffinery import: objects for ${commit.slice(0, 12)} (${i + 1}/${batches.length})`])).stdout.trim();
390+ const size = batch.reduce((s, b) => s + b.size, 0);
391+ while (!(await this.pushTemp(carrier, size, label))) {
392+ /* budget shrank; the carrier is retried as is (a lone object can't be split further) */
393+ }
394+ }
395+ }
396+
397+ /** Push one chunk to a temporary ref. Returns false when the budget was lowered and the caller should re-plan. */
398+ private async pushTemp(oid: string, size: number, label: string): Promise<boolean> {
399+ if (this.haveSet.has(oid)) return true;
400+ const ref = `${TEMP_PREFIX}c${++this.temp}`;
401+ const n = this.result.chunks + 1;
402+ await this.o.progress({ phase: "push", message: `Pushing history of ${label.replace(/^refs\/(heads|tags)\//, "")}: chunk ${n} (${fmtBytes(size)})`, chunks: this.result.chunks, bytes: this.result.bytes });
403+ const out = await this.push([`${oid}:${ref}`]);
404+ if (out.code === 0 && !parsePorcelain(out.stdout).some((s) => !s.ok)) {
405+ this.tempRefs.push(ref);
406+ this.addHave(oid);
407+ this.result.chunks++;
408+ this.result.bytes += size;
409+ return true;
410+ }
411+ const text = `${out.stdout}\n${out.stderr}`;
412+ if (isSecretRejection(text)) {
413+ for (const f of parseSecretFindings(out.stderr)) this.findings.set(`${f.kind}:${f.blob}`, f);
414+ const found = await this.findingsWithPaths();
415+ const what = found.map((f) => `${f.kind} in ${f.path ?? `blob ${f.blob}`}`).join("; ");
416+ throw new SecretsFound(
417+ this.spec.secrets === "bypass"
418+ ? `Push protection found possible secrets in the repository's history (${what}), and only a human admin of the target repository can import them with a bypass.`
419+ : `Push protection found possible secrets in the repository's history (${what}). The import was stopped as requested; rotate the secrets, or import again with the bypass option.`,
420+ found,
421+ );
422+ }
423+ if (isSizeRejection(text)) {
424+ if (this.budget / 2 >= MIN_BUDGET) {
425+ this.budget = Math.floor(this.budget / 2);
426+ this.o.log(`push of ${fmtBytes(size)} was rejected (${lastLine(out.stderr)}); retrying with chunks of ${fmtBytes(this.budget)}`);
427+ this.temp--;
428+ return false;
429+ }
430+ // Chunking can no longer help: a single object is larger than the push limit and cannot be split.
431+ throw new ImportError(`a single Git object is ${fmtBytes(size)}, larger than the 100 MB push limit, so it cannot be imported. Remove the oversized file from the repository's history (e.g. with git filter-repo) and import again.`);
432+ }
433+ throw new ImportError(`push failed: ${lastLine(out.stderr) || lastLine(out.stdout)}`);
434+ }
435+
436+ /** `git push` to Diffinery in import mode; collects push protection findings from the output. */
437+ async push(refspecs: string[]): Promise<{ code: number; stdout: string; stderr: string }> {
438+ const out = await this.git.run(["push", "--porcelain", "--no-verify", ...this.pushOpts, this.spec.target.pushUrl, ...refspecs], { allowFail: true });
439+ if (/Push protection bypassed/i.test(out.stderr)) for (const f of parseSecretFindings(out.stderr)) this.findings.set(`${f.kind}:${f.blob}`, f);
440+ return out;
441+ }
442+
443+ async deleteTempRefs(): Promise<void> {
444+ for (let i = 0; i < this.tempRefs.length; i += REF_BATCH) {
445+ const batch = this.tempRefs.slice(i, i + REF_BATCH);
446+ const out = await this.push(batch.map((r) => `:${r}`));
447+ if (out.code !== 0) this.result.warnings.push(`Some temporary import refs could not be removed: ${lastLine(out.stderr)}`);
448+ }
449+ this.tempRefs = [];
450+ }
451+
452+ /** Findings with a path where each blob appears, so people know what to rotate. */
453+ async findingsWithPaths(): Promise<ImportSecretFinding[]> {
454+ const out: ImportSecretFinding[] = [];
455+ for (const f of [...this.findings.values()].slice(0, 100)) {
456+ const full = (await this.git.run(["rev-parse", "--verify", "--quiet", `${f.blob}^{blob}`], { allowFail: true })).stdout.trim() || f.blob;
457+ let where: string | undefined = f.path;
458+ if (!where && /^[0-9a-f]{40}$/.test(full)) {
459+ const raw = await this.git.run(["log", "--all", "-n", "1", `--find-object=${full}`, "--raw", "--no-abbrev", "--format="], { allowFail: true });
460+ const line = raw.stdout.split("\n").find((l) => l.includes(full));
461+ where = line?.split("\t").slice(-1)[0];
462+ }
463+ out.push({ kind: f.kind, blob: full, path: where });
464+ }
465+ return out;
466+ }
467+}
468+
469+function fmtBytes(n: number): string {
470+ if (n < 1024) return `${n} B`;
471+ if (n < 1024 * 1024) return `${(n / 1024).toFixed(1)} KB`;
472+ return `${(n / 1024 / 1024).toFixed(1)} MB`;
473+}
modifiedcontainers/runner/src/main.ts+14−0View file
@@ -7,11 +7,15 @@
77 * agent.mjs --sandbox AI-agent sandbox HTTP server on :8080
88 * agent.mjs --registry-relay HTTPS proxy for the Docker daemon that splits big
99 * image layer uploads to the Diffinery registry (see relay.ts)
10+ *
11+ * With DIFFINERY_IMPORT_ID and DIFFINERY_IMPORT_TOKEN set (instead of a job), it
12+ * runs a GitHub import's Git transfer (see importer.ts) and exits.
1013 */
1114 import { readFile } from "node:fs/promises";
1215 import type { JobSpec } from "@diffinery/shared";
1316 import { apiBaseFor, runAgentJob } from "./agent.js";
1417 import { RunnerApi } from "./api.js";
18+import { runImportAgent } from "./importer.js";
1519 import { startSandbox } from "./sandbox.js";
1620 import { startRelay } from "./relay.js";
1721
@@ -76,6 +80,16 @@
7680 return;
7781 }
7882
83+ if (env.DIFFINERY_IMPORT_ID) {
84+ const api = env.DIFFINERY_API_URL;
85+ const token = env.DIFFINERY_IMPORT_TOKEN;
86+ if (!api || !token) fail("DIFFINERY_API_URL and DIFFINERY_IMPORT_TOKEN are required in import mode");
87+ const chunk = Number(env.DIFFINERY_IMPORT_CHUNK_BYTES);
88+ const result = await runImportAgent({ apiUrl: api, importId: env.DIFFINERY_IMPORT_ID, token, workRoot, chunkBytes: Number.isFinite(chunk) && chunk > 0 ? chunk : undefined });
89+ process.stdout.write(`[diffinery-import] ${result.ok ? "finished" : `failed: ${result.error}`}\n`);
90+ process.exit(result.ok ? 0 : 1);
91+ }
92+
7993 let spec: JobSpec;
8094 let apiBase: string | null;
8195 if (args.spec) {
addedcontainers/runner/test/import.test.ts+141−0View file
@@ -1,0 +1,141 @@
1+import { execFileSync } from "node:child_process";
2+import { randomBytes } from "node:crypto";
3+import { chmodSync, mkdtempSync, readFileSync, writeFileSync } from "node:fs";
4+import { tmpdir } from "node:os";
5+import path from "node:path";
6+import { describe, expect, it } from "vitest";
7+import type { ImportSpec } from "@diffinery/shared";
8+import { batchBySize, furthestWithin, importDestination, importRefKind, isSecretRejection, isSizeRejection, parsePorcelain, parseSecretFindings, sortRefsForImport } from "../src/import-plan.js";
9+import { transfer } from "../src/importer.js";
10+
11+describe("import chunk planning", () => {
12+ it("finds the furthest commit within budget with few probes", async () => {
13+ const sizes = [10, 25, 40, 70, 90, 140, 200];
14+ const probes: number[] = [];
15+ const at = async (i: number) => (probes.push(i), sizes[i]!);
16+ expect(await furthestWithin(sizes.length, at, 100)).toEqual({ index: 4, size: 90 });
17+ expect(probes.length).toBeLessThanOrEqual(4);
18+ expect(await furthestWithin(sizes.length, at, 1000)).toEqual({ index: 6, size: 200 });
19+ expect(await furthestWithin(sizes.length, at, 5)).toBeNull();
20+ expect(await furthestWithin(0, at, 100)).toBeNull();
21+ // Exactly on the budget counts as fitting.
22+ expect(await furthestWithin(sizes.length, at, 40)).toEqual({ index: 2, size: 40 });
23+ });
24+
25+ it("batches objects by size, keeping oversized ones alone", () => {
26+ const items = [{ size: 30 }, { size: 30 }, { size: 50 }, { size: 500 }, { size: 10 }];
27+ expect(batchBySize(items, 100).map((b) => b.map((x) => x.size))).toEqual([[30, 30], [50], [500], [10]]);
28+ expect(batchBySize([], 100)).toEqual([]);
29+ });
30+
31+ it("orders refs: default branch, branches, tags, then pull heads by number", () => {
32+ const refs = ["refs/pull/10/head", "refs/tags/v1", "refs/heads/zeta", "refs/pull/2/head", "refs/heads/main", "refs/heads/alpha"];
33+ expect(sortRefsForImport(refs, "main")).toEqual(["refs/heads/main", "refs/heads/alpha", "refs/heads/zeta", "refs/tags/v1", "refs/pull/2/head", "refs/pull/10/head"]);
34+ expect(importRefKind("refs/pull/3/merge")).toBeNull();
35+ expect(importRefKind("refs/notes/commits")).toBeNull();
36+ expect(importRefKind("refs/pull/3/head")).toBe("pull");
37+ expect(importRefKind("refs/heads/gh-readonly-queue/main/pr-3-abc")).toBeNull();
38+ expect(importDestination("refs/pull/3/head")).toBe("refs/diffinery-import/pull/3");
39+ expect(importDestination("refs/heads/main")).toBe("refs/heads/main");
40+ });
41+
42+ it("reads push protection and size rejections from git output", () => {
43+ const out = "remote: Push protection bypassed by alice: Imported from GitHub o/r\nremote: AWS access key in blob 0123456789\nremote: Private key in blob abcdef0123\n";
44+ expect(parseSecretFindings(out)).toEqual([
45+ { kind: "AWS access key", blob: "0123456789" },
46+ { kind: "Private key", blob: "abcdef0123" },
47+ ]);
48+ expect(isSecretRejection("! [remote rejected] main -> main (secret detected (push protection))")).toBe(true);
49+ expect(isSizeRejection("remote: push exceeds 100 MB; split it into smaller pushes")).toBe(true);
50+ expect(isSizeRejection("error: RPC failed; HTTP 413 curl 22")).toBe(true);
51+ expect(isSizeRejection("! [rejected] main (non-fast-forward)")).toBe(false);
52+ expect(parsePorcelain("To x\n*\trefs/heads/a:refs/heads/a\t[new branch]\n!\trefs/heads/b:refs/heads/b\t[remote rejected] (invalid ref name)\nDone\n")).toEqual([
53+ { ref: "refs/heads/a", ok: true, reason: "[new branch]" },
54+ { ref: "refs/heads/b", ok: false, reason: "invalid ref name" },
55+ ]);
56+ });
57+});
58+
59+describe("import transfer (real git)", () => {
60+ const env = { ...process.env, GIT_AUTHOR_NAME: "T", GIT_AUTHOR_EMAIL: "t@example.com", GIT_COMMITTER_NAME: "T", GIT_COMMITTER_EMAIL: "t@example.com", GIT_CONFIG_GLOBAL: "/dev/null", GIT_CONFIG_NOSYSTEM: "1" };
61+ const g = (cwd: string, ...args: string[]) => execFileSync("git", args, { cwd, env, stdio: "pipe" }).toString().trim();
62+
63+ it("pushes history in chunks under a size limit the server enforces, then every ref, and cleans up", async () => {
64+ const root = mkdtempSync(path.join(tmpdir(), "diffinery-import-"));
65+ const work = path.join(root, "work");
66+ g(root, "init", "-q", "-b", "main", work);
67+ for (let i = 0; i < 12; i++) {
68+ writeFileSync(path.join(work, `f${i}.bin`), randomBytes(40 * 1024));
69+ g(work, "add", "-A");
70+ g(work, "commit", "-q", "-m", `c${i}`);
71+ }
72+ // A side branch merged back, and one commit too large for any single push.
73+ g(work, "checkout", "-q", "-b", "side", "HEAD~6");
74+ for (let i = 0; i < 4; i++) {
75+ writeFileSync(path.join(work, `s${i}.bin`), randomBytes(40 * 1024));
76+ g(work, "add", "-A");
77+ g(work, "commit", "-q", "-m", `s${i}`);
78+ }
79+ g(work, "checkout", "-q", "main");
80+ g(work, "merge", "-q", "--no-ff", "-m", "merge side", "side");
81+ for (let i = 0; i < 5; i++) writeFileSync(path.join(work, `big${i}.bin`), randomBytes(60 * 1024));
82+ g(work, "add", "-A");
83+ g(work, "commit", "-q", "-m", "big");
84+ g(work, "tag", "-a", "v1", "-m", "release 1");
85+ g(work, "tag", "light");
86+ const source = path.join(root, "source.git");
87+ g(root, "clone", "-q", "--bare", work, source);
88+ g(source, "update-ref", "refs/pull/7/head", g(work, "rev-parse", "side"));
89+ g(source, "update-ref", "refs/pull/7/merge", g(work, "rev-parse", "main"));
90+ g(source, "update-ref", "refs/heads/gh-readonly-queue/main/pr-7-abc", g(work, "rev-parse", "main"));
91+
92+ // The "server": a bare repository whose pre-receive hook refuses packs over 150 KB.
93+ const target = path.join(root, "target.git");
94+ g(root, "init", "-q", "--bare", target);
95+ g(target, "config", "receive.advertisePushOptions", "true");
96+ const hook = path.join(target, "hooks", "pre-receive");
97+ const sizes = path.join(root, "sizes.log");
98+ writeFileSync(
99+ hook,
100+ `#!/bin/sh
101+size=$(du -sb "$GIT_QUARANTINE_PATH" 2>/dev/null | cut -f1)
102+echo "$size $GIT_PUSH_OPTION_0" >> "${sizes}"
103+if [ "\${size:-0}" -gt 153600 ]; then echo "push exceeds 100 MB; split it into smaller pushes"; exit 1; fi
104+`,
105+ );
106+ chmodSync(hook, 0o755);
107+
108+ const spec: ImportSpec = {
109+ importId: "t1",
110+ source: { fullName: "octo/repo", cloneUrl: `file://${source}`, defaultBranch: "main" },
111+ githubToken: null,
112+ target: { fullName: "me/repo", pushUrl: `file://${target}` },
113+ pushToken: "dfy_test",
114+ lfs: false,
115+ secrets: "bypass",
116+ bypassReason: "Imported from GitHub octo/repo",
117+ // Deliberately larger than what the server accepts: the agent must split further.
118+ chunkBytes: 1024 * 1024,
119+ };
120+ const phases: string[] = [];
121+ const result = await transfer(spec, { workRoot: path.join(root, "agent"), progress: async (p) => void phases.push(p.phase), log: () => {} });
122+ expect(result.ok).toBe(true);
123+ expect(result.warnings).toEqual([]);
124+ expect(result).toMatchObject({ branches: 2, tags: 2, pullRefs: 1, defaultBranch: "main" });
125+ expect(result.chunks).toBeGreaterThan(3);
126+ expect(phases).toContain("push");
127+
128+ const refsOf = (dir: string) => g(dir, "for-each-ref", "--format=%(refname) %(objectname)").split("\n").sort();
129+ // Pull request heads wait in the import namespace for the server; merge refs and merge queue branches are skipped.
130+ const expected = refsOf(source)
131+ .filter((l) => !l.startsWith("refs/pull/7/merge") && !l.startsWith("refs/heads/gh-readonly-queue/"))
132+ .map((l) => l.replace("refs/pull/7/head", "refs/diffinery-import/pull/7"))
133+ .sort();
134+ expect(refsOf(target)).toEqual(expected);
135+ g(target, "fsck", "--no-progress");
136+ // Every accepted push carried the import option, and the oversized ones were refused, then split.
137+ const log = readFileSync(sizes, "utf8").trim().split("\n");
138+ expect(log.every((l) => l.endsWith("diffinery-import"))).toBe(true);
139+ expect(log.some((l) => Number(l.split(" ")[0]) > 153600)).toBe(true);
140+ });
141+});
addedpackages/shared/src/imports.ts+63−0View file
@@ -1,0 +1,63 @@
1+/**
2+ * Contract between the Diffinery control plane (Worker) and the runner agent's
3+ * import mode, which copies a repository's Git data (and Git LFS objects) from
4+ * GitHub into Diffinery inside a Cloudflare Container.
5+ *
6+ * The agent calls `${apiUrl}/api/v3/diffinery/imports/:id/agent/...` with
7+ * `Authorization: Bearer <push token>`: a short-lived Diffinery token scoped to
8+ * the target repository, which the Worker revokes when the transfer ends.
9+ */
10+
11+export interface ImportSpec {
12+ importId: string;
13+ /** GitHub side. `cloneUrl` is checked by the Worker to be on the GitHub host being imported from. */
14+ source: { fullName: string; cloneUrl: string; defaultBranch: string };
15+ /** Read token for GitHub (private repositories, LFS); null for public repositories. */
16+ githubToken: string | null;
17+ /** Diffinery side: the repository's git URL, e.g. https://diffinery.com/acme/app.git */
18+ target: { fullName: string; pushUrl: string };
19+ /** The short-lived push token (same as the bearer token used for these calls). */
20+ pushToken: string;
21+ /** Git LFS objects are copied too. */
22+ lfs: boolean;
23+ /** Push protection findings in old history: push with an audited admin bypass, or stop. */
24+ secrets: "bypass" | "stop";
25+ /** Reason sent with `-o allow-secrets=` when `secrets` is "bypass". */
26+ bypassReason: string;
27+ /** Upper bound for one push request body; the agent splits history to stay under it. */
28+ chunkBytes: number;
29+}
30+
31+export type ImportPhase = "clone" | "lfs-fetch" | "push" | "refs" | "lfs-push" | "cleanup" | "done";
32+
33+export interface ImportProgress {
34+ phase: ImportPhase;
35+ message: string;
36+ /** Pushed history chunks so far, and the bytes they carried. */
37+ chunks?: number;
38+ bytes?: number;
39+}
40+
41+export interface ImportSecretFinding {
42+ kind: string;
43+ /** Blob id, or its abbreviation as the server reported it. */
44+ blob: string;
45+ /** A path where the blob appears in history, when the agent could find one. */
46+ path?: string;
47+}
48+
49+export interface ImportGitResult {
50+ ok: boolean;
51+ error?: string;
52+ branches: number;
53+ tags: number;
54+ pullRefs: number;
55+ chunks: number;
56+ bytes: number;
57+ lfsObjects: number;
58+ lfsBytes: number;
59+ /** The GitHub default branch exists in the import. */
60+ defaultBranch: string | null;
61+ secretFindings: ImportSecretFinding[];
62+ warnings: string[];
63+}
modifiedpackages/shared/src/index.ts+1−0View file
@@ -1,2 +1,3 @@
11 export * from "./runner.js";
22 export * from "./events.js";
3+export * from "./imports.js";