README, advisory agent approvals, cross-site request guard
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AizKXSsengXZRegJxRATkL
9 files changed+825−6
Files (9)
addedREADME.md+157−0View file
| @@ -1,0 +1,157 @@ | ||
| 1 | +# Diffinery | |
| 2 | + | |
| 3 | +A git forge for teams of people and AI agents, running entirely on Cloudflare. | |
| 4 | +Git hosting, pull requests and reviews, issues, GitHub-Actions-compatible CI, | |
| 5 | +an MCP server for agents, and sandboxes, with no servers to run. | |
| 6 | + | |
| 7 | +- **Drop-in for GitHub workflows.** `.github/workflows/*.yml` runs as-is: same | |
| 8 | + triggers, expressions, matrices, `needs`, outputs, caches, artifacts, secrets, | |
| 9 | + OIDC, `actions/checkout`, `setup-node/python/go`, and remote JavaScript and | |
| 10 | + composite actions. | |
| 11 | +- **Built for agents.** Agents are first-class accounts owned by a person or | |
| 12 | + team, with their own tokens and limits. They can't push or merge to protected | |
| 13 | + branches, and their pull requests always need a human approval. Every commit | |
| 14 | + they make is stamped with who they act for. They work through MCP | |
| 15 | + (`/mcp`) or the REST API, and can start sandboxes to build and test. | |
| 16 | +- **Secure by default.** Cloudflare Access handles sign-in, while invites, | |
| 17 | + roles and branch protection decide what people can do. Pushes are scanned for | |
| 18 | + leaked credentials. Secrets use envelope encryption, and every sensitive | |
| 19 | + action goes to an append-only audit log. | |
| 20 | + | |
| 21 | +## How it fits on Cloudflare | |
| 22 | + | |
| 23 | +| Piece | Cloudflare product | | |
| 24 | +|---|---| | |
| 25 | +| HTTP, git smart HTTP, REST API, MCP, web UI | Workers (Hono) | | |
| 26 | +| One repository: refs, objects, PRs, issues, reviews, checks, runs | Durable Object with SQLite (`RepoDO`), one per repository | | |
| 27 | +| Large objects, Git LFS, logs, caches, artifacts, event archive | R2 | | |
| 28 | +| Accounts, orgs, tokens, invites, notifications, audit log | D1 | | |
| 29 | +| Webhooks, notifications, audit fan-out | Queues (with a dead-letter queue) | | |
| 30 | +| Workflow runs (durable, survive restarts, 6 h job timeout) | Workflows | | |
| 31 | +| Job runners and agent sandboxes | Containers (Ubuntu 24.04 image in `containers/runner`) | | |
| 32 | +| Scheduled workflows (`on: schedule`) | Cron Triggers | | |
| 33 | +| Live run logs and PR updates | Durable Object WebSockets (hibernation) | | |
| 34 | +| Sign-in | Cloudflare Access (GitHub, Google, email one-time PIN, etc.) | | |
| 35 | + | |
| 36 | +``` | |
| 37 | +apps/diffinery the Worker (src/index.ts), D1 migrations, wrangler config, e2e test | |
| 38 | +apps/cli the `diffinery` command-line client | |
| 39 | +containers/runner runner agent + sandbox server, and the container image | |
| 40 | +packages/git-core git objects, packfiles, deltas, diff/merge (pure TypeScript) | |
| 41 | +packages/workflow-yaml GitHub Actions workflow parser, expressions, matrix, cron | |
| 42 | +packages/shared types shared by the Worker and the runner | |
| 43 | +scripts bootstrap.sh, ensure-secrets.sh, render-config.mjs | |
| 44 | +``` | |
| 45 | + | |
| 46 | +## Deploying | |
| 47 | + | |
| 48 | +Deploys run from GitHub Actions on every push to `main` (`.github/workflows/deploy.yml`). | |
| 49 | +The workflow needs two repository secrets: | |
| 50 | + | |
| 51 | +- `CLOUDFLARE_API_TOKEN`: account permissions Workers Scripts, D1, R2, | |
| 52 | + Queues, Workflows, Containers and Access: Apps and Policies (all Edit), and | |
| 53 | + zone permission Workers Routes Edit on `diffinery.com`. | |
| 54 | +- `CLOUDFLARE_ACCOUNT_ID` | |
| 55 | + | |
| 56 | +On each deploy, `scripts/bootstrap.sh` creates anything that is missing: the D1 | |
| 57 | +database, R2 bucket, queues and the two Access applications (sign-in for the | |
| 58 | +site, and a bypass for the git, API, MCP and runner paths, which check | |
| 59 | +Diffinery tokens themselves). It then applies D1 migrations. `wrangler deploy` | |
| 60 | +builds and pushes the runner image. On the first deploy, `ensure-secrets.sh` | |
| 61 | +generates `MASTER_KEY` and the OIDC signing key inside the job, so neither | |
| 62 | +value is ever printed. | |
| 63 | + | |
| 64 | +If the token can't read your Zero Trust organization, set a repository | |
| 65 | +**variable** `ACCESS_TEAM_DOMAIN` (for example `codician.cloudflareaccess.com`). | |
| 66 | + | |
| 67 | +To let people sign in with GitHub, add GitHub as a login method under Zero | |
| 68 | +Trust → Settings → Authentication. Email one-time PIN works without any setup. | |
| 69 | + | |
| 70 | +## Inviting people | |
| 71 | + | |
| 72 | +1. The emails in `BOOTSTRAP_ADMINS` (in `apps/diffinery/wrangler.jsonc`) | |
| 73 | + become site admins the first time they sign in. | |
| 74 | +2. Anyone else who signs in lands on a waiting page. They can request access, | |
| 75 | + which admins approve, or an admin or org owner can send an invite from | |
| 76 | + *Settings → People* or with `diffinery invite friend@example.com --org codician`. | |
| 77 | +3. Repository and org roles (read, triage, write, maintain, admin) decide what | |
| 78 | + each person can do. | |
| 79 | + | |
| 80 | +Nobody needs a Cloudflare account. They need an Access login and, for git and | |
| 81 | +the CLI, a personal token from *Settings → Tokens*. | |
| 82 | + | |
| 83 | +```sh | |
| 84 | +git clone https://diffinery.com/codician/website.git # username: anything, password: your dfy_ token | |
| 85 | +diffinery login && diffinery auth setup-git # or let the CLI handle credentials | |
| 86 | +``` | |
| 87 | + | |
| 88 | +## Agents | |
| 89 | + | |
| 90 | +Create an agent under *Settings → Agents* or with `diffinery agent create reviewer-bot --org codician`. | |
| 91 | +Then point any MCP client at it: | |
| 92 | + | |
| 93 | +```json | |
| 94 | +{ "mcpServers": { "diffinery": { "type": "http", "url": "https://diffinery.com/mcp", "headers": { "Authorization": "Bearer dfy_…" } } } } | |
| 95 | +``` | |
| 96 | + | |
| 97 | +Tools include `get_file`, `list_tree`, `commit_files` (no git client needed), | |
| 98 | +`create_branch`, `create_pull_request` (with an `intent` for reviewers), | |
| 99 | +`review_pull_request`, `merge_pull_request` (policy-guarded), issues, | |
| 100 | +`list_workflow_runs` and `get_job_logs`, `rerun_workflow`, and | |
| 101 | +`sandbox_create`/`sandbox_exec` for running builds and tests. | |
| 102 | + | |
| 103 | +Agent guardrails apply everywhere: git, REST, MCP and the web UI. | |
| 104 | + | |
| 105 | +- An agent's role is capped at write, and never exceeds its owner's role. | |
| 106 | +- Agents can't push to the default branch or protected branches, and can't | |
| 107 | + merge there. Turn this on per branch with `agentsMayPush` or `agentsMayMerge`. | |
| 108 | +- A pull request opened by an agent needs at least one human approval, and | |
| 109 | + approvals from agents don't count toward it. | |
| 110 | +- Agents can't change branch protection or read or set secrets. | |
| 111 | + | |
| 112 | +## Workflows | |
| 113 | + | |
| 114 | +Put workflow files in `.github/workflows/` (or `.diffinery/workflows/`). | |
| 115 | +Supported triggers are `push`, `pull_request`, `workflow_dispatch` (with | |
| 116 | +inputs) and `schedule`, with branch, tag and path filters. Supported workflow | |
| 117 | +features: | |
| 118 | + | |
| 119 | +- `needs` and job outputs, `if` with `success()`, `failure()`, `always()` and | |
| 120 | + `cancelled()`, `strategy.matrix` with include and exclude, `fail-fast` and | |
| 121 | + `max-parallel`, `concurrency` with `cancel-in-progress`, and `timeout-minutes`. | |
| 122 | +- `continue-on-error`, `defaults.run`, `env` at every level, `$GITHUB_OUTPUT`, | |
| 123 | + `$GITHUB_ENV`, `$GITHUB_PATH`, `$GITHUB_STEP_SUMMARY`, and masking of | |
| 124 | + secrets and `::add-mask::` values. | |
| 125 | +- `secrets.*`, `vars.*`, `GITHUB_TOKEN` (a per-job token), and | |
| 126 | + `permissions: id-token: write` for OIDC (issuer `https://diffinery.com`). | |
| 127 | +- `runs-on: ubuntu-latest` (and any `ubuntu-*` or `linux` label) runs in a | |
| 128 | + Cloudflare Container. `runs-on: [self-hosted, …]` jobs wait for a runner you | |
| 129 | + start with `diffinery runner -R owner/repo --labels gpu,macos`. Use that for | |
| 130 | + macOS, GPUs or private networks. | |
| 131 | + | |
| 132 | +**Not supported yet:** reusable workflows (`jobs.<id>.uses`), job | |
| 133 | +`container:` and `services:`, Docker container actions, `actions/github-script`, | |
| 134 | +Windows and macOS hosted runners, `pull_request_target`, `merge_group`, and | |
| 135 | +`deepen-not` / partial clone filters. Pull request runs use the head commit | |
| 136 | +rather than a test merge commit. A job that uses an unsupported feature fails | |
| 137 | +with a clear message instead of misbehaving. | |
| 138 | + | |
| 139 | +## Developing | |
| 140 | + | |
| 141 | +```sh | |
| 142 | +pnpm install | |
| 143 | +pnpm -r build && pnpm -r typecheck && pnpm -r test | |
| 144 | +pnpm --filter @diffinery/app db:migrate:local | |
| 145 | +pnpm dev # http://localhost:8787, signed in as dev@diffinery.local | |
| 146 | +pnpm --filter @diffinery/app e2e # wrangler dev + real git + the runner agent | |
| 147 | +``` | |
| 148 | + | |
| 149 | +The end-to-end test does the following: | |
| 150 | + | |
| 151 | +1. Pushes a repository with real `git`. | |
| 152 | +2. Runs its CI through the actual runner agent. | |
| 153 | +3. Checks secret scanning and branch protection. | |
| 154 | +4. Has an agent open a pull request that needs a human approval and a green | |
| 155 | + check, then merges it. | |
| 156 | +5. Clones shallow, then deepens the clone. | |
| 157 | +6. Drives the MCP server. |
addedapps/cli/src/commands/migrate.ts+296−0View file
| @@ -1,0 +1,296 @@ | ||
| 1 | +import { mkdtemp, rm } from "node:fs/promises"; | |
| 2 | +import { tmpdir } from "node:os"; | |
| 3 | +import path from "node:path"; | |
| 4 | +import { ApiError, type Api } from "../api.js"; | |
| 5 | +import { parseRepoArg, repoPath, str, type Command, type Ctx, type Group, type RepoRef } from "../cli.js"; | |
| 6 | +import { basicAuthHeader, git, gitCapture, withGitConfig } from "../git.js"; | |
| 7 | +import { GitHub, GitHubError } from "../github.js"; | |
| 8 | +import { CliError } from "../io.js"; | |
| 9 | +import type { IssueJson } from "./pr.js"; | |
| 10 | +import type { RepoJson } from "./repo.js"; | |
| 11 | + | |
| 12 | +interface GhRepo { | |
| 13 | + full_name: string; | |
| 14 | + name: string; | |
| 15 | + private: boolean; | |
| 16 | + visibility?: "public" | "private" | "internal"; | |
| 17 | + description: string | null; | |
| 18 | + default_branch: string; | |
| 19 | + clone_url: string; | |
| 20 | + html_url: string; | |
| 21 | +} | |
| 22 | + | |
| 23 | +interface GhUser { | |
| 24 | + login: string; | |
| 25 | +} | |
| 26 | + | |
| 27 | +interface GhIssue { | |
| 28 | + number: number; | |
| 29 | + title: string; | |
| 30 | + body: string | null; | |
| 31 | + state: "open" | "closed"; | |
| 32 | + state_reason?: string | null; | |
| 33 | + user: GhUser | null; | |
| 34 | + labels: Array<{ name: string } | string>; | |
| 35 | + comments: number; | |
| 36 | + created_at: string; | |
| 37 | + pull_request?: unknown; | |
| 38 | +} | |
| 39 | + | |
| 40 | +interface GhPull { | |
| 41 | + number: number; | |
| 42 | + title: string; | |
| 43 | + body: string | null; | |
| 44 | + draft?: boolean; | |
| 45 | + user: GhUser | null; | |
| 46 | + created_at: string; | |
| 47 | + head: { ref: string; repo: { full_name: string } | null }; | |
| 48 | + base: { ref: string }; | |
| 49 | +} | |
| 50 | + | |
| 51 | +interface GhComment { | |
| 52 | + body: string | null; | |
| 53 | + user: GhUser | null; | |
| 54 | + created_at: string; | |
| 55 | +} | |
| 56 | + | |
| 57 | +const day = (iso: string) => iso.slice(0, 10); | |
| 58 | +const who = (u: GhUser | null) => (u ? `@${u.login}` : "a deleted user"); | |
| 59 | + | |
| 60 | +export function attribution(kind: "issue" | "pull request" | "comment", n: number, user: GhUser | null, createdAt: string, htmlBase: string): string { | |
| 61 | + const link = `${htmlBase}/${kind === "pull request" ? "pull" : "issues"}/${n}`; | |
| 62 | + if (kind === "comment") return `> Imported from GitHub: comment by ${who(user)} on ${day(createdAt)}`; | |
| 63 | + return `> Imported from GitHub ${kind} [#${n}](${link}) by ${who(user)} on ${day(createdAt)}`; | |
| 64 | +} | |
| 65 | + | |
| 66 | +const withAttribution = (line: string, body: string | null) => `${line}\n\n${body ?? ""}`.trimEnd(); | |
| 67 | + | |
| 68 | +interface Summary { | |
| 69 | + repo: string; | |
| 70 | + created: boolean; | |
| 71 | + branches: number; | |
| 72 | + tags: number; | |
| 73 | + pushFailures: string[]; | |
| 74 | + defaultBranch: string | null; | |
| 75 | + labels: number; | |
| 76 | + issues: Array<{ from: number; to: number }>; | |
| 77 | + pulls: Array<{ from: number; to: number }>; | |
| 78 | + skippedPulls: Array<{ number: number; reason: string }>; | |
| 79 | + comments: number; | |
| 80 | + variables: string[]; | |
| 81 | + secrets: string[]; | |
| 82 | + warnings: string[]; | |
| 83 | +} | |
| 84 | + | |
| 85 | +async function optional<T>(ctx: Ctx, s: Summary, what: string, f: () => Promise<T>): Promise<T | null> { | |
| 86 | + try { | |
| 87 | + return await f(); | |
| 88 | + } catch (e) { | |
| 89 | + if (e instanceof GitHubError && (e.status === 403 || e.status === 404)) { | |
| 90 | + s.warnings.push(`could not read ${what} from GitHub (${e.status}; the token may lack access)`); | |
| 91 | + ctx.err(`${ctx.c.yellow("warning:")} could not read ${what} from GitHub (${e.status})`); | |
| 92 | + return null; | |
| 93 | + } | |
| 94 | + throw e; | |
| 95 | + } | |
| 96 | +} | |
| 97 | + | |
| 98 | +async function ensureRepo(ctx: Ctx, api: Api, to: RepoRef, me: string, gh: GhRepo, forcePrivate: boolean): Promise<{ repo: RepoJson; created: boolean }> { | |
| 99 | + const existing = await api.raw<RepoJson>("GET", repoPath(to), { allow: [404] }); | |
| 100 | + if (existing.status !== 404) { | |
| 101 | + ctx.out(`Using existing repository ${to.owner}/${to.name}`); | |
| 102 | + return { repo: existing.data, created: false }; | |
| 103 | + } | |
| 104 | + const visibility = forcePrivate ? "private" : (gh.visibility ?? (gh.private ? "private" : "public")); | |
| 105 | + const body = { name: to.name, description: gh.description ?? undefined, visibility }; | |
| 106 | + const repo = await api.post<RepoJson>(to.owner === me ? "/user/repos" : `/orgs/${encodeURIComponent(to.owner)}/repos`, body); | |
| 107 | + ctx.out(`${ctx.c.green("✓")} Created ${repo.full_name} (${repo.visibility})`); | |
| 108 | + return { repo, created: true }; | |
| 109 | +} | |
| 110 | + | |
| 111 | +async function migrateGit(ctx: Ctx, gh: GhRepo, ghToken: string | null, target: string, token: string | null, diffineryUrl: string, s: Summary): Promise<Set<string>> { | |
| 112 | + const dir = await mkdtemp(path.join(tmpdir(), "diffinery-migrate-")); | |
| 113 | + const mirror = path.join(dir, "repo.git"); | |
| 114 | + try { | |
| 115 | + ctx.out(`Cloning ${gh.html_url} …`); | |
| 116 | + const cloneEnv = withGitConfig(ctx.io.env, ghToken && /^https?:\/\//.test(gh.clone_url) ? [basicAuthHeader(new URL(gh.clone_url).origin, "x-access-token", ghToken)] : []); | |
| 117 | + const cl = await gitCapture(ctx.io, ["clone", "--mirror", "--quiet", gh.clone_url, mirror], { env: { ...cloneEnv, GIT_TERMINAL_PROMPT: "0" } }); | |
| 118 | + if (cl.code !== 0) throw new CliError(`git clone --mirror failed: ${cl.stderr.trim()}`); | |
| 119 | + const refs = (await git(ctx.io, ["for-each-ref", "--format=%(refname)", "refs/heads", "refs/tags"], { cwd: mirror })).split("\n").filter(Boolean); | |
| 120 | + const branches = new Set(refs.filter((r) => r.startsWith("refs/heads/")).map((r) => r.slice("refs/heads/".length))); | |
| 121 | + s.branches = branches.size; | |
| 122 | + s.tags = refs.length - branches.size; | |
| 123 | + if (!refs.length) { | |
| 124 | + ctx.out("The GitHub repository is empty; nothing to push."); | |
| 125 | + return branches; | |
| 126 | + } | |
| 127 | + ctx.out(`Pushing ${s.branches} branch${s.branches === 1 ? "" : "es"} and ${s.tags} tag${s.tags === 1 ? "" : "s"} to ${target} …`); | |
| 128 | + const pushEnv = { ...withGitConfig(ctx.io.env, token ? [basicAuthHeader(diffineryUrl, "x-token", token)] : []), GIT_TERMINAL_PROMPT: "0" }; | |
| 129 | + const remote = `${diffineryUrl}/${target}.git`; | |
| 130 | + const all = await gitCapture(ctx.io, ["push", "--quiet", remote, "refs/heads/*:refs/heads/*", "refs/tags/*:refs/tags/*"], { cwd: mirror, env: pushEnv }); | |
| 131 | + if (all.code !== 0) { | |
| 132 | + // One big push can hit request size limits or a rejected ref; fall back to ref by ref. | |
| 133 | + ctx.err(`${ctx.c.yellow("warning:")} bulk push failed (${all.stderr.trim().split("\n").pop()}); pushing refs one at a time`); | |
| 134 | + for (const ref of refs) { | |
| 135 | + const r = await gitCapture(ctx.io, ["push", "--quiet", remote, `${ref}:${ref}`], { cwd: mirror, env: pushEnv }); | |
| 136 | + if (r.code !== 0) { | |
| 137 | + s.pushFailures.push(`${ref}: ${r.stderr.trim().split("\n").pop()}`); | |
| 138 | + ctx.err(`${ctx.c.red("X")} ${ref}: ${r.stderr.trim().split("\n").pop()}`); | |
| 139 | + } | |
| 140 | + } | |
| 141 | + } | |
| 142 | + return branches; | |
| 143 | + } finally { | |
| 144 | + await rm(dir, { recursive: true, force: true }); | |
| 145 | + } | |
| 146 | +} | |
| 147 | + | |
| 148 | +async function migrateIssues(ctx: Ctx, gh: GitHub, api: Api, src: string, gr: GhRepo, to: RepoRef, branches: Set<string>, s: Summary): Promise<void> { | |
| 149 | + const base = repoPath(to); | |
| 150 | + const ghPath = `/repos/${src}`; | |
| 151 | + // Labels | |
| 152 | + const labels = (await optional(ctx, s, "labels", () => gh.paginate<{ name: string; color: string; description: string | null }>(`${ghPath}/labels`))) ?? []; | |
| 153 | + for (const l of labels) { | |
| 154 | + try { | |
| 155 | + await api.post(`${base}/labels`, { name: l.name, color: l.color, description: l.description ?? "" }); | |
| 156 | + s.labels++; | |
| 157 | + } catch (e) { | |
| 158 | + s.warnings.push(`label ${l.name}: ${(e as Error).message}`); | |
| 159 | + } | |
| 160 | + } | |
| 161 | + if (labels.length) ctx.out(`${ctx.c.green("✓")} Copied ${s.labels} label${s.labels === 1 ? "" : "s"}`); | |
| 162 | + | |
| 163 | + const openPulls = new Map<number, GhPull>(); | |
| 164 | + for (const p of (await optional(ctx, s, "pull requests", () => gh.paginate<GhPull>(`${ghPath}/pulls`, { state: "open" }))) ?? []) openPulls.set(p.number, p); | |
| 165 | + const items = ((await optional(ctx, s, "issues", () => gh.paginate<GhIssue>(`${ghPath}/issues`, { state: "all", sort: "created", direction: "asc" }))) ?? []).sort((a, b) => a.number - b.number); | |
| 166 | + ctx.out(`Importing ${items.length} issues and pull requests …`); | |
| 167 | + | |
| 168 | + for (const it of items) { | |
| 169 | + let created: IssueJson | null = null; | |
| 170 | + if (it.pull_request) { | |
| 171 | + const p = openPulls.get(it.number); | |
| 172 | + if (it.state !== "open" || !p) { | |
| 173 | + s.skippedPulls.push({ number: it.number, reason: it.state === "open" ? "not found" : "closed or merged" }); | |
| 174 | + continue; | |
| 175 | + } | |
| 176 | + if (p.head.repo?.full_name?.toLowerCase() !== gr.full_name.toLowerCase()) { | |
| 177 | + s.skippedPulls.push({ number: it.number, reason: "head branch is in a fork" }); | |
| 178 | + continue; | |
| 179 | + } | |
| 180 | + if (!branches.has(p.head.ref) || !branches.has(p.base.ref)) { | |
| 181 | + s.skippedPulls.push({ number: it.number, reason: `branch ${branches.has(p.head.ref) ? p.base.ref : p.head.ref} missing` }); | |
| 182 | + continue; | |
| 183 | + } | |
| 184 | + try { | |
| 185 | + created = await api.post<IssueJson>(`${base}/pulls`, { title: p.title, body: withAttribution(attribution("pull request", p.number, p.user, p.created_at, gr.html_url), p.body), head: p.head.ref, base: p.base.ref, draft: p.draft || undefined }); | |
| 186 | + s.pulls.push({ from: it.number, to: created.number }); | |
| 187 | + } catch (e) { | |
| 188 | + s.skippedPulls.push({ number: it.number, reason: (e as Error).message }); | |
| 189 | + continue; | |
| 190 | + } | |
| 191 | + } else { | |
| 192 | + const labelNames = it.labels.map((l) => (typeof l === "string" ? l : l.name)).filter(Boolean); | |
| 193 | + const body = withAttribution(attribution("issue", it.number, it.user, it.created_at, gr.html_url), it.body); | |
| 194 | + try { | |
| 195 | + created = await api.post<IssueJson>(`${base}/issues`, { title: it.title, body, labels: labelNames.length ? labelNames : undefined }); | |
| 196 | + } catch (e) { | |
| 197 | + if (!(e instanceof ApiError) || !labelNames.length) throw e; | |
| 198 | + created = await api.post<IssueJson>(`${base}/issues`, { title: it.title, body }); | |
| 199 | + s.warnings.push(`issue #${it.number}: labels not applied (${e.message})`); | |
| 200 | + } | |
| 201 | + s.issues.push({ from: it.number, to: created.number }); | |
| 202 | + } | |
| 203 | + if (it.comments > 0) { | |
| 204 | + const comments = (await optional(ctx, s, `comments of #${it.number}`, () => gh.paginate<GhComment>(`${ghPath}/issues/${it.number}/comments`))) ?? []; | |
| 205 | + for (const cm of comments) { | |
| 206 | + await api.post(`${base}/issues/${created.number}/comments`, { body: withAttribution(attribution("comment", it.number, cm.user, cm.created_at, gr.html_url), cm.body) }); | |
| 207 | + s.comments++; | |
| 208 | + } | |
| 209 | + } | |
| 210 | + if (it.state === "closed" && !it.pull_request) { | |
| 211 | + await api.patch(`${base}/issues/${created.number}`, { state: "closed", state_reason: it.state_reason === "not_planned" ? "not_planned" : "completed" }); | |
| 212 | + } | |
| 213 | + ctx.out(ctx.c.dim(` GitHub #${it.number} → #${created.number} ${it.title.slice(0, 60)}`)); | |
| 214 | + } | |
| 215 | +} | |
| 216 | + | |
| 217 | +const github: Command = { | |
| 218 | + name: "migrate github", | |
| 219 | + summary: "Import a GitHub repository (git data, and optionally issues)", | |
| 220 | + args: "<GH-OWNER/GH-REPO>", | |
| 221 | + description: `Creates the Diffinery repository if needed, mirrors all branches and tags (not refs/pull), | |
| 222 | +sets the default branch, copies Actions variables and lists Actions secret names to set. | |
| 223 | +With --issues, also copies labels, issues and comments (with attribution lines), and recreates | |
| 224 | +open pull requests whose branches were imported. | |
| 225 | + | |
| 226 | +GitHub access uses GH_TOKEN or GITHUB_TOKEN (needed for private repositories; GITHUB_API_URL | |
| 227 | +selects a GitHub Enterprise API).`, | |
| 228 | + options: { | |
| 229 | + to: { type: "string", short: "t", desc: "Diffinery repository to create or push into (default: <you>/<gh-repo>)", arg: "OWNER/NAME" }, | |
| 230 | + issues: { type: "boolean", short: "i", desc: "Also import labels, issues, comments and open pull requests" }, | |
| 231 | + private: { type: "boolean", short: "p", desc: "Create the repository as private regardless of GitHub's visibility" }, | |
| 232 | + }, | |
| 233 | + json: true, | |
| 234 | + minArgs: 1, | |
| 235 | + maxArgs: 1, | |
| 236 | + async run(ctx, o, [srcArg]) { | |
| 237 | + const src = parseRepoArg(srcArg!.replace(/^https?:\/\/github\.com\//, "")); | |
| 238 | + const srcName = `${src.owner}/${src.name}`; | |
| 239 | + const api = await ctx.authedApi(); | |
| 240 | + const cfg = await ctx.config(); | |
| 241 | + const ghToken = ctx.io.env.GH_TOKEN || ctx.io.env.GITHUB_TOKEN || null; | |
| 242 | + const gh = new GitHub({ baseUrl: ctx.io.env.GITHUB_API_URL, token: ghToken, sleep: (ms) => ctx.io.sleep(ms), log: (m) => ctx.err(ctx.c.yellow(m)) }); | |
| 243 | + if (!ghToken) ctx.err(ctx.c.yellow("note: GH_TOKEN/GITHUB_TOKEN is not set; only public data can be read and GitHub rate limits are low")); | |
| 244 | + | |
| 245 | + const me = await api.get<{ login: string }>("/user"); | |
| 246 | + const gr = await gh.get<GhRepo>(`/repos/${srcName}`); | |
| 247 | + const to = str(o, "to") ? parseRepoArg(str(o, "to")!) : { owner: me.login, name: gr.name }; | |
| 248 | + const target = `${to.owner}/${to.name}`; | |
| 249 | + const s: Summary = { repo: target, created: false, branches: 0, tags: 0, pushFailures: [], defaultBranch: null, labels: 0, issues: [], pulls: [], skippedPulls: [], comments: 0, variables: [], secrets: [], warnings: [] }; | |
| 250 | + | |
| 251 | + const { created } = await ensureRepo(ctx, api, to, me.login, gr, o.private === true); | |
| 252 | + s.created = created; | |
| 253 | + const branches = await migrateGit(ctx, gr, ghToken, target, cfg.token, cfg.url, s); | |
| 254 | + if (branches.has(gr.default_branch)) { | |
| 255 | + await api.patch(repoPath(to), { default_branch: gr.default_branch }); | |
| 256 | + s.defaultBranch = gr.default_branch; | |
| 257 | + } | |
| 258 | + | |
| 259 | + const vars = (await optional(ctx, s, "Actions variables", () => gh.paginate<{ name: string; value: string }>(`/repos/${srcName}/actions/variables`, {}, "variables"))) ?? []; | |
| 260 | + for (const v of vars) { | |
| 261 | + await api.put(`${repoPath(to)}/actions/variables/${encodeURIComponent(v.name)}`, { value: v.value }); | |
| 262 | + s.variables.push(v.name); | |
| 263 | + } | |
| 264 | + s.secrets = ((await optional(ctx, s, "Actions secret names", () => gh.paginate<{ name: string }>(`/repos/${srcName}/actions/secrets`, {}, "secrets"))) ?? []).map((x) => x.name); | |
| 265 | + | |
| 266 | + if (o.issues) await migrateIssues(ctx, gh, api, srcName, gr, to, branches, s); | |
| 267 | + | |
| 268 | + if (ctx.json) { | |
| 269 | + ctx.out(JSON.stringify(s, null, 2)); | |
| 270 | + return s.pushFailures.length ? 1 : 0; | |
| 271 | + } | |
| 272 | + const c = ctx.c; | |
| 273 | + ctx.out(""); | |
| 274 | + ctx.out(c.bold(`Imported ${srcName} → ${target}`)); | |
| 275 | + ctx.out(` git: ${s.branches} branches, ${s.tags} tags${s.pushFailures.length ? c.red(` (${s.pushFailures.length} refs failed)`) : ""}`); | |
| 276 | + ctx.out(` default: ${s.defaultBranch ?? c.dim("unchanged")}`); | |
| 277 | + ctx.out(` variables: ${s.variables.length}`); | |
| 278 | + if (o.issues) { | |
| 279 | + ctx.out(` labels: ${s.labels}`); | |
| 280 | + ctx.out(` issues: ${s.issues.length} (${s.comments} comments)`); | |
| 281 | + ctx.out(` pulls: ${s.pulls.length} recreated, ${s.skippedPulls.filter((p) => p.reason !== "closed or merged").length} skipped`); | |
| 282 | + for (const p of s.skippedPulls.filter((x) => x.reason !== "closed or merged")) ctx.out(c.dim(` #${p.number}: ${p.reason}`)); | |
| 283 | + } | |
| 284 | + if (s.secrets.length) { | |
| 285 | + ctx.out(""); | |
| 286 | + ctx.out(`GitHub does not reveal secret values. Set these ${s.secrets.length} secret${s.secrets.length === 1 ? "" : "s"}:`); | |
| 287 | + for (const n of s.secrets) ctx.out(` diffinery secret set ${n} -R ${target}`); | |
| 288 | + } | |
| 289 | + for (const w of s.warnings) ctx.err(`${c.yellow("warning:")} ${w}`); | |
| 290 | + ctx.out(""); | |
| 291 | + ctx.out(`${cfg.url}/${target}`); | |
| 292 | + return s.pushFailures.length ? 1 : 0; | |
| 293 | + }, | |
| 294 | +}; | |
| 295 | + | |
| 296 | +export const migrateGroup: Group = { name: "migrate", summary: "Import repositories from other forges", commands: [github] }; |
addedapps/cli/src/commands/runner.ts+169−0View file
| @@ -1,0 +1,169 @@ | ||
| 1 | +import { spawn, type ChildProcess } from "node:child_process"; | |
| 2 | +import { existsSync } from "node:fs"; | |
| 3 | +import { mkdir } from "node:fs/promises"; | |
| 4 | +import { homedir, hostname } from "node:os"; | |
| 5 | +import path from "node:path"; | |
| 6 | +import { createInterface } from "node:readline"; | |
| 7 | +import { ApiError, type Api } from "../api.js"; | |
| 8 | +import { list, parseRepoArg, repoPath, str, type Command, type Ctx, type RepoRef } from "../cli.js"; | |
| 9 | +import { CliError, type Io } from "../io.js"; | |
| 10 | + | |
| 11 | +export interface ClaimedJob { | |
| 12 | + job_id: string; | |
| 13 | + run_id: string; | |
| 14 | + name: string; | |
| 15 | + token: string; | |
| 16 | + api_url: string; | |
| 17 | +} | |
| 18 | + | |
| 19 | +const AGENT_HELP = `The runner needs the Diffinery runner agent (a single Node.js file). Build it from the | |
| 20 | +Diffinery repository with: | |
| 21 | + | |
| 22 | + pnpm --filter @diffinery/runner build # produces containers/runner/dist/agent.mjs | |
| 23 | + | |
| 24 | +then pass --agent /path/to/agent.mjs or set DIFFINERY_RUNNER_AGENT.`; | |
| 25 | + | |
| 26 | +export function resolveAgentPath(io: Io, flag: string | undefined): string { | |
| 27 | + const p = flag ?? io.env.DIFFINERY_RUNNER_AGENT; | |
| 28 | + if (!p) throw new CliError(`no runner agent given.\n\n${AGENT_HELP}`); | |
| 29 | + const abs = path.resolve(io.cwd, p); | |
| 30 | + if (!existsSync(abs)) throw new CliError(`runner agent not found at ${abs}.\n\n${AGENT_HELP}`); | |
| 31 | + return abs; | |
| 32 | +} | |
| 33 | + | |
| 34 | +/** POST .../actions/runners/claim: a job, or null when nothing is waiting (204). */ | |
| 35 | +export async function claim(api: Api, repo: RepoRef, name: string, labels: string[]): Promise<ClaimedJob | null> { | |
| 36 | + const res = await api.raw<ClaimedJob | null>("POST", `${repoPath(repo)}/actions/runners/claim`, { body: { name, labels } }); | |
| 37 | + if (res.status === 204 || !res.data || typeof res.data !== "object") return null; | |
| 38 | + return res.data; | |
| 39 | +} | |
| 40 | + | |
| 41 | +/** Environment for the agent: the job's credentials, minus the CLI's own token. */ | |
| 42 | +export function agentEnv(base: NodeJS.ProcessEnv, job: ClaimedJob, workRoot: string, echo: boolean): NodeJS.ProcessEnv { | |
| 43 | + const env: NodeJS.ProcessEnv = { ...base }; | |
| 44 | + for (const k of ["DIFFINERY_TOKEN", "DIFFINERY_URL", "DIFFINERY_RUNNER_AGENT", "GH_TOKEN", "GITHUB_TOKEN"]) delete env[k]; | |
| 45 | + env.DIFFINERY_API_URL = job.api_url; | |
| 46 | + env.DIFFINERY_JOB_ID = job.job_id; | |
| 47 | + env.DIFFINERY_JOB_TOKEN = job.token; | |
| 48 | + env.DIFFINERY_WORK_ROOT ??= workRoot; | |
| 49 | + if (echo) env.DIFFINERY_ECHO_LOGS = "1"; | |
| 50 | + return env; | |
| 51 | +} | |
| 52 | + | |
| 53 | +function pipeLines(stream: NodeJS.ReadableStream, prefix: string, write: (s: string) => void): Promise<void> { | |
| 54 | + return new Promise((resolve) => { | |
| 55 | + const rl = createInterface({ input: stream, crlfDelay: Infinity }); | |
| 56 | + rl.on("line", (l) => write(`${prefix}${l}\n`)); | |
| 57 | + rl.on("close", () => resolve()); | |
| 58 | + }); | |
| 59 | +} | |
| 60 | + | |
| 61 | +async function runJob(ctx: Ctx, agent: string, job: ClaimedJob, repo: RepoRef, workRoot: string, echo: boolean, onChild: (c: ChildProcess | null) => void): Promise<number> { | |
| 62 | + const c = ctx.c; | |
| 63 | + const prefix = c.dim(`[${job.name}] `); | |
| 64 | + ctx.out(`${c.cyan("→")} ${repo.owner}/${repo.name}: running job ${c.bold(job.name)} ${c.dim(`(run ${job.run_id}, job ${job.job_id})`)}`); | |
| 65 | + await mkdir(workRoot, { recursive: true }); | |
| 66 | + const child = spawn(process.execPath, [agent], { cwd: workRoot, env: agentEnv(ctx.io.env, job, workRoot, echo), stdio: ["ignore", "pipe", "pipe"] }); | |
| 67 | + onChild(child); | |
| 68 | + const done = Promise.all([pipeLines(child.stdout!, prefix, (s) => ctx.io.stdout.write(s)), pipeLines(child.stderr!, prefix, (s) => ctx.io.stderr.write(s))]); | |
| 69 | + const code = await new Promise<number>((resolve) => { | |
| 70 | + child.on("error", (e) => { | |
| 71 | + ctx.err(`${c.red("error:")} could not start the agent: ${e.message}`); | |
| 72 | + resolve(1); | |
| 73 | + }); | |
| 74 | + child.on("exit", (code, signal) => resolve(code ?? (signal ? 128 + (signal === "SIGKILL" ? 9 : 15) : 1))); | |
| 75 | + }); | |
| 76 | + await done; | |
| 77 | + onChild(null); | |
| 78 | + ctx.out(`${code === 0 ? c.green("✓") : c.red("X")} job ${job.name} finished${code === 0 ? "" : ` with exit code ${code}`}`); | |
| 79 | + return code; | |
| 80 | +} | |
| 81 | + | |
| 82 | +export const runnerCommand: Command = { | |
| 83 | + name: "runner", | |
| 84 | + summary: "Run self-hosted workflow jobs on this machine", | |
| 85 | + description: `Polls the given repositories for jobs with 'runs-on: [self-hosted, ...]' whose labels this | |
| 86 | +runner offers, and runs each with the Diffinery runner agent. Requires admin access to the | |
| 87 | +repositories. Jobs run directly on this machine with your user's permissions. | |
| 88 | + | |
| 89 | +${AGENT_HELP}`, | |
| 90 | + options: { | |
| 91 | + repo: { type: "string", short: "R", multiple: true, desc: "Repository to take jobs from (default: the git origin)", arg: "OWNER/NAME" }, | |
| 92 | + labels: { type: "string", short: "l", desc: "Extra labels this runner offers, comma separated ('self-hosted' is implied)", arg: "a,b" }, | |
| 93 | + name: { type: "string", short: "n", desc: "Runner name (default: host name)", arg: "NAME" }, | |
| 94 | + once: { type: "boolean", desc: "Run at most one job, then exit with its exit code" }, | |
| 95 | + agent: { type: "string", short: "a", desc: "Path to the runner agent (agent.mjs); default DIFFINERY_RUNNER_AGENT", arg: "PATH" }, | |
| 96 | + "work-dir": { type: "string", short: "w", desc: "Working directory for jobs (default ~/.cache/diffinery/runner/<name>)", arg: "DIR" }, | |
| 97 | + interval: { type: "string", desc: "Seconds between polls when idle (default 5)", arg: "SECONDS" }, | |
| 98 | + quiet: { type: "boolean", short: "q", desc: "Do not echo step output locally (logs still go to the server)" }, | |
| 99 | + }, | |
| 100 | + maxArgs: 0, | |
| 101 | + async run(ctx, o) { | |
| 102 | + const agent = resolveAgentPath(ctx.io, str(o, "agent")); | |
| 103 | + const repos = list(o, "repo").length ? list(o, "repo").map(parseRepoArg) : [await ctx.repo()]; | |
| 104 | + const labels = (str(o, "labels") ?? "").split(",").map((s) => s.trim()).filter(Boolean); | |
| 105 | + const name = str(o, "name") ?? hostname(); | |
| 106 | + const intervalMs = Math.max(0, Number(str(o, "interval") ?? 5)) * 1000; | |
| 107 | + if (!Number.isFinite(intervalMs)) throw new CliError("--interval must be a number", 2); | |
| 108 | + const cacheHome = ctx.io.env.XDG_CACHE_HOME && path.isAbsolute(ctx.io.env.XDG_CACHE_HOME) ? ctx.io.env.XDG_CACHE_HOME : path.join(ctx.io.env.HOME || homedir(), ".cache"); | |
| 109 | + const workRoot = path.resolve(ctx.io.cwd, str(o, "work-dir") ?? path.join(cacheHome, "diffinery", "runner", name.replace(/[^A-Za-z0-9._-]/g, "_"))); | |
| 110 | + const api = await ctx.authedApi(); | |
| 111 | + const c = ctx.c; | |
| 112 | + | |
| 113 | + let stopping = false; | |
| 114 | + let child: ChildProcess | null = null; | |
| 115 | + const abort = new AbortController(); | |
| 116 | + let signals = 0; | |
| 117 | + const onSignal = () => { | |
| 118 | + signals++; | |
| 119 | + stopping = true; | |
| 120 | + abort.abort(); | |
| 121 | + if (child && child.exitCode === null) { | |
| 122 | + if (signals === 1) { | |
| 123 | + ctx.err(c.yellow("Stopping: asking the running job to cancel (press Ctrl-C again to kill it)…")); | |
| 124 | + child.kill("SIGTERM"); | |
| 125 | + } else child.kill("SIGKILL"); | |
| 126 | + } | |
| 127 | + }; | |
| 128 | + process.on("SIGINT", onSignal); | |
| 129 | + process.on("SIGTERM", onSignal); | |
| 130 | + | |
| 131 | + try { | |
| 132 | + ctx.out(`Runner ${c.bold(name)} offering [${["self-hosted", ...labels].join(", ")}] for ${repos.map((r) => `${r.owner}/${r.name}`).join(", ")}`); | |
| 133 | + if (!o.once) ctx.out(c.dim("Waiting for jobs. Press Ctrl-C to stop.")); | |
| 134 | + let lastError = ""; | |
| 135 | + while (!stopping) { | |
| 136 | + let ran = false; | |
| 137 | + for (const repo of repos) { | |
| 138 | + if (stopping) break; | |
| 139 | + let job: ClaimedJob | null; | |
| 140 | + try { | |
| 141 | + job = await claim(api, repo, name, labels); | |
| 142 | + lastError = ""; | |
| 143 | + } catch (e) { | |
| 144 | + if (o.once || (e instanceof ApiError && [401, 403, 404].includes(e.status))) throw e; | |
| 145 | + const msg = (e as Error).message; | |
| 146 | + if (msg !== lastError) ctx.err(`${c.yellow("warning:")} ${msg} (will retry)`); | |
| 147 | + lastError = msg; | |
| 148 | + continue; | |
| 149 | + } | |
| 150 | + if (!job) continue; | |
| 151 | + ran = true; | |
| 152 | + const code = await runJob(ctx, agent, job, repo, workRoot, !o.quiet, (ch) => (child = ch)); | |
| 153 | + if (o.once) return code; | |
| 154 | + if (stopping) break; | |
| 155 | + } | |
| 156 | + if (o.once) { | |
| 157 | + ctx.out("No jobs waiting."); | |
| 158 | + return 0; | |
| 159 | + } | |
| 160 | + if (!ran && !stopping) await ctx.io.sleep(intervalMs, abort.signal); | |
| 161 | + } | |
| 162 | + ctx.out("Runner stopped."); | |
| 163 | + return 0; | |
| 164 | + } finally { | |
| 165 | + process.off("SIGINT", onSignal); | |
| 166 | + process.off("SIGTERM", onSignal); | |
| 167 | + } | |
| 168 | + }, | |
| 169 | +}; |
addedapps/cli/src/github.ts+127−0View file
| @@ -1,0 +1,127 @@ | ||
| 1 | +import { CliError } from "./io.js"; | |
| 2 | +import { VERSION } from "./version.js"; | |
| 3 | + | |
| 4 | +export class GitHubError extends CliError { | |
| 5 | + constructor( | |
| 6 | + readonly status: number, | |
| 7 | + message: string, | |
| 8 | + ) { | |
| 9 | + super(message); | |
| 10 | + } | |
| 11 | +} | |
| 12 | + | |
| 13 | +export interface GitHubOptions { | |
| 14 | + baseUrl?: string; | |
| 15 | + token?: string | null; | |
| 16 | + sleep: (ms: number) => Promise<void>; | |
| 17 | + log?: (msg: string) => void; | |
| 18 | + /** Longest single wait for a rate-limit reset before giving up (default 1h). */ | |
| 19 | + maxWaitMs?: number; | |
| 20 | + now?: () => number; | |
| 21 | +} | |
| 22 | + | |
| 23 | +/** Next-page URL from a GitHub `Link` header. */ | |
| 24 | +export function nextLink(link: string | null): string | null { | |
| 25 | + if (!link) return null; | |
| 26 | + for (const part of link.split(",")) { | |
| 27 | + const m = /<([^>]+)>\s*;\s*rel="?next"?/.exec(part); | |
| 28 | + if (m) return m[1]!; | |
| 29 | + } | |
| 30 | + return null; | |
| 31 | +} | |
| 32 | + | |
| 33 | +/** Minimal GitHub REST client with pagination and rate-limit handling. */ | |
| 34 | +export class GitHub { | |
| 35 | + readonly baseUrl: string; | |
| 36 | + constructor(private o: GitHubOptions) { | |
| 37 | + this.baseUrl = (o.baseUrl ?? "https://api.github.com").replace(/\/+$/, ""); | |
| 38 | + } | |
| 39 | + | |
| 40 | + private async fetchRetrying(url: string): Promise<Response> { | |
| 41 | + const now = this.o.now ?? Date.now; | |
| 42 | + const maxWait = this.o.maxWaitMs ?? 3600_000; | |
| 43 | + for (let attempt = 0; ; attempt++) { | |
| 44 | + const headers: Record<string, string> = { accept: "application/vnd.github+json", "x-github-api-version": "2022-11-28", "user-agent": `diffinery-cli/${VERSION}` }; | |
| 45 | + if (this.o.token) headers.authorization = `Bearer ${this.o.token}`; | |
| 46 | + let res: Response; | |
| 47 | + try { | |
| 48 | + res = await fetch(url, { headers }); | |
| 49 | + } catch (e) { | |
| 50 | + if (attempt < 3) { | |
| 51 | + await this.o.sleep(1000 * 2 ** attempt); | |
| 52 | + continue; | |
| 53 | + } | |
| 54 | + throw new CliError(`could not reach GitHub (${new URL(url).host}): ${(e as Error).message}`); | |
| 55 | + } | |
| 56 | + if ((res.status === 403 || res.status === 429) && attempt < 8) { | |
| 57 | + const retryAfter = res.headers.get("retry-after"); | |
| 58 | + const remaining = res.headers.get("x-ratelimit-remaining"); | |
| 59 | + const reset = res.headers.get("x-ratelimit-reset"); | |
| 60 | + let wait: number | null = null; | |
| 61 | + if (retryAfter && Number.isFinite(Number(retryAfter))) wait = Number(retryAfter) * 1000; | |
| 62 | + else if (remaining === "0" && reset && Number.isFinite(Number(reset))) wait = Math.max(0, Number(reset) * 1000 - now()) + 1000; | |
| 63 | + else if (res.status === 429) wait = 60_000; | |
| 64 | + else { | |
| 65 | + const text = await res.clone().text(); | |
| 66 | + if (/secondary rate limit|abuse/i.test(text)) wait = 60_000; | |
| 67 | + } | |
| 68 | + if (wait !== null) { | |
| 69 | + if (wait > maxWait) throw new GitHubError(res.status, `GitHub rate limit: reset is ${Math.ceil(wait / 60000)} minutes away; try again later${this.o.token ? "" : " or set GH_TOKEN"}`); | |
| 70 | + this.o.log?.(`GitHub rate limit reached; waiting ${Math.ceil(wait / 1000)}s…`); | |
| 71 | + await res.body?.cancel(); | |
| 72 | + await this.o.sleep(wait); | |
| 73 | + continue; | |
| 74 | + } | |
| 75 | + } | |
| 76 | + if (res.status >= 500 && attempt < 3) { | |
| 77 | + await res.body?.cancel(); | |
| 78 | + await this.o.sleep(1000 * 2 ** attempt); | |
| 79 | + continue; | |
| 80 | + } | |
| 81 | + return res; | |
| 82 | + } | |
| 83 | + } | |
| 84 | + | |
| 85 | + private url(path: string, query?: Record<string, string | number>): string { | |
| 86 | + const u = new URL(/^https?:\/\//.test(path) ? path : `${this.baseUrl}${path}`); | |
| 87 | + for (const [k, v] of Object.entries(query ?? {})) u.searchParams.set(k, String(v)); | |
| 88 | + return u.toString(); | |
| 89 | + } | |
| 90 | + | |
| 91 | + private async check(res: Response, url: string): Promise<void> { | |
| 92 | + if (res.ok) return; | |
| 93 | + let msg = res.statusText; | |
| 94 | + try { | |
| 95 | + const b = (await res.json()) as { message?: string }; | |
| 96 | + if (b.message) msg = b.message; | |
| 97 | + } catch { | |
| 98 | + /* not JSON */ | |
| 99 | + } | |
| 100 | + throw new GitHubError(res.status, `GitHub ${new URL(url).pathname}: ${res.status} ${msg}`); | |
| 101 | + } | |
| 102 | + | |
| 103 | + async get<T>(path: string, query?: Record<string, string | number>): Promise<T> { | |
| 104 | + const url = this.url(path, query); | |
| 105 | + const res = await this.fetchRetrying(url); | |
| 106 | + await this.check(res, url); | |
| 107 | + return (await res.json()) as T; | |
| 108 | + } | |
| 109 | + | |
| 110 | + /** | |
| 111 | + * All pages of a list endpoint. `key` picks the array out of wrapped responses | |
| 112 | + * such as `{ total_count, variables: [...] }`. | |
| 113 | + */ | |
| 114 | + async paginate<T>(path: string, query: Record<string, string | number> = {}, key?: string): Promise<T[]> { | |
| 115 | + const out: T[] = []; | |
| 116 | + let url: string | null = this.url(path, { per_page: 100, ...query }); | |
| 117 | + while (url) { | |
| 118 | + const res = await this.fetchRetrying(url); | |
| 119 | + await this.check(res, url); | |
| 120 | + const body = (await res.json()) as unknown; | |
| 121 | + const items = key ? ((body as Record<string, unknown>)[key] as T[] | undefined) : (body as T[]); | |
| 122 | + if (Array.isArray(items)) out.push(...items); | |
| 123 | + url = nextLink(res.headers.get("link")); | |
| 124 | + } | |
| 125 | + return out; | |
| 126 | + } | |
| 127 | +} |
addedapps/cli/src/main.ts+52−0View file
| @@ -1,0 +1,52 @@ | ||
| 1 | +import { dispatch, type Registry } from "./cli.js"; | |
| 2 | +import { adminCommands, agentGroup, secretGroup, tokenGroup, variableGroup } from "./commands/admin.js"; | |
| 3 | +import { authCommands, authGroup } from "./commands/auth.js"; | |
| 4 | +import { issueGroup } from "./commands/issue.js"; | |
| 5 | +import { migrateGroup } from "./commands/migrate.js"; | |
| 6 | +import { prGroup } from "./commands/pr.js"; | |
| 7 | +import { repoGroup } from "./commands/repo.js"; | |
| 8 | +import { runGroup } from "./commands/run.js"; | |
| 9 | +import { runnerCommand } from "./commands/runner.js"; | |
| 10 | +import { workflowGroup } from "./commands/workflow.js"; | |
| 11 | +import { CliError, processIo, type Io } from "./io.js"; | |
| 12 | + | |
| 13 | +export const registry: Registry = { | |
| 14 | + groups: [authGroup, repoGroup, prGroup, issueGroup, runGroup, workflowGroup, secretGroup, variableGroup, tokenGroup, agentGroup, migrateGroup], | |
| 15 | + commands: [...authCommands, ...adminCommands, runnerCommand], | |
| 16 | +}; | |
| 17 | + | |
| 18 | +/** Runs the CLI and returns its exit code (never calls process.exit). */ | |
| 19 | +export async function main(argv: string[], io: Io = processIo()): Promise<number> { | |
| 20 | + try { | |
| 21 | + return await dispatch(registry, argv, io); | |
| 22 | + } catch (e) { | |
| 23 | + if (e instanceof CliError) { | |
| 24 | + io.stderr.write(`error: ${e.message}\n`); | |
| 25 | + return e.code; | |
| 26 | + } | |
| 27 | + io.stderr.write(`error: ${e instanceof Error ? (e.stack ?? e.message) : String(e)}\n`); | |
| 28 | + return 1; | |
| 29 | + } | |
| 30 | +} | |
| 31 | + | |
| 32 | +const isEntry = (() => { | |
| 33 | + const entry = process.argv[1]; | |
| 34 | + if (!entry) return false; | |
| 35 | + return /(?:^|[\\/])(?:diffinery(?:\.mjs)?|main\.[cm]?[jt]s)$/.test(entry); | |
| 36 | +})(); | |
| 37 | + | |
| 38 | +if (isEntry) { | |
| 39 | + // EPIPE when piping into `head` and similar should end quietly. | |
| 40 | + process.stdout.on("error", (e: NodeJS.ErrnoException) => { | |
| 41 | + if (e.code === "EPIPE") process.exit(0); | |
| 42 | + }); | |
| 43 | + main(process.argv.slice(2)).then( | |
| 44 | + (code) => { | |
| 45 | + process.exitCode = code; | |
| 46 | + }, | |
| 47 | + (e: unknown) => { | |
| 48 | + process.stderr.write(`error: ${String(e)}\n`); | |
| 49 | + process.exitCode = 1; | |
| 50 | + }, | |
| 51 | + ); | |
| 52 | +} |
modifiedapps/diffinery/src/app.ts+11−1View file
| @@ -2,7 +2,7 @@ | ||
| 2 | 2 | import type { DiffineryEvent } from "@diffinery/shared"; |
| 3 | 3 | import { cronMatches } from "@diffinery/workflow-yaml"; |
| 4 | 4 | import type { Env } from "./env.js"; |
| 5 | -import { resolvePrincipal, type AppVars } from "./auth/principal.js"; | |
| 5 | +import { credentialFrom, resolvePrincipal, type AppVars } from "./auth/principal.js"; | |
| 6 | 6 | import { Directory } from "./lib/directory.js"; |
| 7 | 7 | import { hmacSha256Hex, newId } from "./lib/util.js"; |
| 8 | 8 | import { errorResponse, loadRepo, repoStub } from "./http/common.js"; |
| @@ -17,6 +17,16 @@ | ||
| 17 | 17 | |
| 18 | 18 | // Security headers on every response, then identity: tokens, Access JWT, or anonymous. |
| 19 | 19 | app.use("*", async (c, next) => { |
| 20 | + // Browser sessions ride on cookies, so refuse state-changing requests that come from another site. | |
| 21 | + // Token-authenticated calls (git, CLI, MCP, runners) carry no ambient credentials and are exempt. | |
| 22 | + if (!["GET", "HEAD", "OPTIONS"].includes(c.req.method) && !credentialFrom(c.req.raw)) { | |
| 23 | + const site = c.req.header("sec-fetch-site"); | |
| 24 | + const origin = c.req.header("origin"); | |
| 25 | + const ours = new URL(c.env.PUBLIC_URL).origin; | |
| 26 | + if ((site && site !== "same-origin" && site !== "none") || (!site && origin && origin !== ours)) { | |
| 27 | + return c.json({ message: "cross-site request refused" }, 403); | |
| 28 | + } | |
| 29 | + } | |
| 20 | 30 | const dir = new Directory(c.env); |
| 21 | 31 | c.set("dir", dir); |
| 22 | 32 | c.set("principal", await resolvePrincipal(c.req.raw, c.env, dir)); |
modifiedapps/diffinery/src/repo/collab.ts+3−2View file
| @@ -541,12 +541,13 @@ | ||
| 541 | 541 | const latest = new Map<string, Review>(); |
| 542 | 542 | for (const rv of this.reviews(number)) if (rv.state !== "COMMENTED") latest.set(rv.author.id, rv); |
| 543 | 543 | for (const rv of latest.values()) { |
| 544 | - if (rv.state === "APPROVED" && (!rules.dismissStaleReviews || rv.commitId === p.head_sha)) result.approvals++; | |
| 544 | + // Agent reviews are advisory: only people's approvals count toward requirements. | |
| 545 | + if (rv.state === "APPROVED" && rv.author.kind !== "agent" && (!rules.dismissStaleReviews || rv.commitId === p.head_sha)) result.approvals++; | |
| 545 | 546 | if (rv.state === "CHANGES_REQUESTED") result.changesRequested.push(rv.author.login); |
| 546 | 547 | } |
| 547 | 548 | const checks = this.checks(p.head_sha); |
| 548 | 549 | result.checks = checks.map((c) => ({ name: c.name, state: c.status === "completed" ? (c.conclusion ?? "neutral") : c.status })); |
| 549 | - if (r.author_kind === "agent" && result.approvals === 0 && (rules.requiredApprovals ?? 0) === 0) { | |
| 550 | + if (r.author_kind === "agent" && result.requiredApprovals < 1) { | |
| 550 | 551 | // Agent-authored changes always need one human approval unless a rule says otherwise. |
| 551 | 552 | result.requiredApprovals = 1; |
| 552 | 553 | } |
modifiedapps/diffinery/src/web/actions.tsx+5−3View file
| @@ -13,6 +13,8 @@ | ||
| 13 | 13 | |
| 14 | 14 | type Loaded = Awaited<ReturnType<typeof loadRepo>>; |
| 15 | 15 | type LogLine = { seq: number; t: string; step: number; text: string }; |
| 16 | +/** Plain (non-RPC-wrapped) shape of a run, which keeps type checking shallow. */ | |
| 17 | +type RunWithJobs = RunInfo & { jobs: JobInfo[] }; | |
| 16 | 18 | |
| 17 | 19 | const ANSI = /\x1b\[[0-9;?]*[A-Za-z]/g; |
| 18 | 20 | const MAX_LOG_LINES = 20000; |
| @@ -278,7 +280,7 @@ | ||
| 278 | 280 | const { repo, caller, stub } = l; |
| 279 | 281 | const base = repoUrl(repo); |
| 280 | 282 | const runId = prm(c, "runId"); |
| 281 | - const run = await stub.run(runId); | |
| 283 | + const run = (await stub.run(runId)) as unknown as RunWithJobs; | |
| 282 | 284 | const artifacts = await stub.artifacts(runId).catch(() => [] as Array<{ name: string; size: number; createdAt: string }>); |
| 283 | 285 | const groups = new Map<string, JobInfo[]>(); |
| 284 | 286 | for (const j of run.jobs) groups.set(j.key, [...(groups.get(j.key) ?? []), j]); |
| @@ -415,8 +417,8 @@ | ||
| 415 | 417 | const base = repoUrl(repo); |
| 416 | 418 | const runId = prm(c, "runId"); |
| 417 | 419 | const jobId = prm(c, "jobId"); |
| 418 | - const run = await stub.run(runId); | |
| 419 | - const job = await stub.job(jobId); | |
| 420 | + const run = (await stub.run(runId)) as unknown as RunWithJobs; | |
| 421 | + const job = (await stub.job(jobId)) as unknown as JobInfo; | |
| 420 | 422 | if (job.runId !== run.id) throw new HttpError(404, "job not found"); |
| 421 | 423 | const logs = await allLogs(l, jobId); |
| 422 | 424 | const byStep = new Map<number, LogLine[]>(); |
modifiedapps/diffinery/test/e2e/e2e.test.mjs+5−0View file
| @@ -262,6 +262,11 @@ | ||
| 262 | 262 | const agentMerge = await api("PUT", `/repos/${owner}/demo/pulls/1/merge`, { merge_method: "squash" }, agentToken); |
| 263 | 263 | assert.ok([403, 405, 409].includes(agentMerge.status), JSON.stringify(agentMerge)); |
| 264 | 264 | |
| 265 | + // Another agent's approval is advisory and does not unblock the merge. | |
| 266 | + const reviewer = (await ok("POST", "/diffinery/agents", { login: "e2e-reviewer" })).token.token; | |
| 267 | + await ok("POST", `/repos/${owner}/demo/pulls/1/reviews`, { event: "APPROVE", body: "bot says ok" }, reviewer); | |
| 268 | + assert.equal((await ok("GET", `/repos/${owner}/demo/pulls/1/merge`)).mergeable, false); | |
| 269 | + | |
| 265 | 270 | // A human reviews (line comment + approval) and merges. |
| 266 | 271 | await ok("POST", `/repos/${owner}/demo/pulls/1/reviews`, { event: "APPROVE", body: "LGTM", comments: [{ path: "app.txt", line: 4, body: "nice line" }] }); |
| 267 | 272 | m = await ok("GET", `/repos/${owner}/demo/pulls/1/merge`); |