diffinery/README.md
| 1 | # Diffinery |
| 2 | |
| 3 | A git forge for teams of people and AI agents, running entirely on Cloudflare. |
| 4 | Git hosting, pull requests and reviews, issues, GitHub-Actions-compatible CI, |
| 5 | an MCP server for agents, and sandboxes, with no servers to run. |
| 6 | |
| 7 | - **Drop-in for GitHub workflows.** `.github/workflows/*.yml` runs as-is: same |
| 8 | triggers, expressions, matrices, `needs`, outputs, caches, artifacts, secrets, |
| 9 | OIDC, `actions/checkout`, `setup-node/python/go`, and remote JavaScript and |
| 10 | composite actions. |
| 11 | - **Built for agents.** Agents are first-class accounts owned by a person or |
| 12 | team, with their own tokens and limits. They can't push or merge to protected |
| 13 | branches, and their pull requests always need a human approval. Every commit |
| 14 | they make is stamped with who they act for. They work through MCP |
| 15 | (`/mcp`) or the REST API, and can start sandboxes to build and test. |
| 16 | - **Secure by default.** People sign in with passkeys, with no passwords to |
| 17 | leak or phish. Invites, roles and branch protection decide what they can do. Pushes are scanned for |
| 18 | leaked credentials and blocked if they contain any; nothing from a blocked |
| 19 | push is kept. A repository admin (a person, never an agent) can push anyway |
| 20 | with `git push -o allow-secrets=<reason>`, which is audited. A file can opt |
| 21 | out with a `diffinery:allow-secret` comment, for test fixtures. Secrets use |
| 22 | envelope encryption, and every sensitive action goes to an append-only audit log. |
| 23 | |
| 24 | ## Documentation |
| 25 | |
| 26 | The full documentation is public at [diffinery.com/docs](https://diffinery.com/docs): |
| 27 | guides for every feature, plus REST API, CLI and MCP references generated from |
| 28 | this code. For AI agents there's [llms.txt](https://diffinery.com/llms.txt), |
| 29 | [llms-full.txt](https://diffinery.com/llms-full.txt), a Markdown version of |
| 30 | every page (add `.md` to its URL) and an |
| 31 | [OpenAPI document](https://diffinery.com/docs/api/openapi.json). The pages live |
| 32 | in [apps/diffinery/docs](apps/diffinery/docs). |
| 33 | |
| 34 | ## How it fits on Cloudflare |
| 35 | |
| 36 | | Piece | Cloudflare product | |
| 37 | |---|---| |
| 38 | | HTTP, git smart HTTP, REST API, MCP, web UI | Workers (Hono) | |
| 39 | | One repository: refs, objects, PRs, issues, reviews, checks, runs | Durable Object with SQLite (`RepoDO`), one per repository | |
| 40 | | Large objects, Git LFS, container images, logs, caches, artifacts, event archive | R2 | |
| 41 | | Accounts, orgs, tokens, invites, notifications, audit log | D1, read from nearby replicas through the Sessions API (each browser keeps a bookmark so it reads its own writes) | |
| 42 | | Webhooks, notifications, audit fan-out | Queues (with a dead-letter queue) | |
| 43 | | Workflow runs (durable, survive restarts, 6 h job timeout) | Workflows | |
| 44 | | Job runners and agent sandboxes | Containers (Ubuntu 24.04 image in `containers/runner`) | |
| 45 | | Scheduled workflows (`on: schedule`) | Cron Triggers | |
| 46 | | Live run logs and PR updates | Durable Object WebSockets (hibernation) | |
| 47 | | Sign-in | Passkeys (WebAuthn) and sessions in D1. Cloudflare Access guards only `/auth/access`, the email recovery path | |
| 48 | |
| 49 | ``` |
| 50 | apps/diffinery the Worker (src/index.ts), D1 migrations, wrangler config, e2e test |
| 51 | apps/cli the `diffinery` command-line client |
| 52 | containers/runner runner agent + sandbox server, and the container image |
| 53 | packages/git-core git objects, packfiles, deltas, diff/merge (pure TypeScript) |
| 54 | packages/workflow-yaml GitHub Actions workflow parser, expressions, matrix, cron |
| 55 | packages/shared types shared by the Worker and the runner |
| 56 | scripts bootstrap.sh, ensure-secrets.sh, render-config.mjs |
| 57 | ``` |
| 58 | |
| 59 | ## Deploying |
| 60 | |
| 61 | Diffinery deploys itself: this repository lives at |
| 62 | `diffinery.com/codician-team/diffinery`, and every push to `main` there runs |
| 63 | CI (`.github/workflows/ci.yml`: typecheck, unit tests and the Worker bundle) on |
| 64 | Diffinery's hosted runners. When it passes, `.github/workflows/deploy.yml` |
| 65 | deploys that commit, followed by the live test (`.github/workflows/live-test.yml`); |
| 66 | the end-to-end suite (`.github/workflows/e2e.yml`) runs alongside. GitHub stays the upstream for |
| 67 | now: `.github/workflows/sync-to-diffinery.yml` copies every branch and tag, and |
| 68 | the secrets and variables below, to Diffinery on each push, then waits for the |
| 69 | Actions runs there and fails if any of them fail. Running Deploy by hand on |
| 70 | GitHub still deploys, as a fallback. |
| 71 | |
| 72 | The workflow needs two repository secrets: |
| 73 | |
| 74 | - `CLOUDFLARE_API_TOKEN`: account permissions Workers Scripts, D1, R2, |
| 75 | Queues, Workflows, Containers and Access: Apps and Policies (all Edit), and |
| 76 | zone permission Workers Routes Edit on `diffinery.com`. |
| 77 | - `CLOUDFLARE_ACCOUNT_ID` |
| 78 | |
| 79 | On each deploy, `scripts/bootstrap.sh` creates anything that is missing: the D1 |
| 80 | database, R2 bucket, queues and one Access application that covers only |
| 81 | `/auth/access` (email-based account recovery). It then applies D1 migrations. `wrangler deploy` |
| 82 | builds and pushes the runner image. On the first deploy, `ensure-secrets.sh` |
| 83 | generates `MASTER_KEY` and the OIDC signing key inside the job, so neither |
| 84 | value is ever printed. |
| 85 | |
| 86 | ### Storage |
| 87 | |
| 88 | Durable and transient data live under separate R2 prefixes, because lifecycle |
| 89 | rules can only select by prefix (`apps/diffinery/src/lib/storage-keys.ts`): |
| 90 | |
| 91 | | Prefix | What | Lifecycle rule | |
| 92 | |---|---|---| |
| 93 | | `repos/` | git objects, LFS objects, release assets | none: kept until the repository is deleted | |
| 94 | | `registry/` | container images | none: unreferenced blobs are swept by the cron job | |
| 95 | | `actions-cache/` | Actions caches | none: Diffinery evicts unused entries after 7 days, and the least recently used above 10 GB per repository | |
| 96 | | `artifacts/` | workflow artifacts | deleted after 90 days (shorter `retention-days` are handled by Diffinery) | |
| 97 | | `logs/` | archived job logs | deleted after 90 days, like GitHub's log retention | |
| 98 | | `actions-staging/` | uploads in progress | deleted after a day | |
| 99 | | `archives/` | zipball and tarball cache | deleted after 30 days (rebuilt when asked for again) | |
| 100 | | `events/` | the event archive | deleted after 400 days | |
| 101 | |
| 102 | Unfinished multipart uploads are aborted after a day everywhere. `bootstrap.sh` |
| 103 | applies these rules on every deploy and stops the deploy if that fails, or if |
| 104 | any rule would still delete objects under `repos/`. Artifacts, caches and logs |
| 105 | written before this layout (under `repos/<id>/...`) are still read from there. |
| 106 | Deleting a repository deletes its objects under the Actions and archive |
| 107 | prefixes. |
| 108 | |
| 109 | If the token can't read your Zero Trust organization, set a repository |
| 110 | **variable** `ACCESS_TEAM_DOMAIN` (for example `codician.cloudflareaccess.com`). |
| 111 | |
| 112 | To run your own Diffinery on another domain, set `DIFFINERY_DOMAIN` (for |
| 113 | example `git.example.com`) when running `scripts/bootstrap.sh`. The Worker's |
| 114 | route, `PUBLIC_URL` and the email sender follow it. Set `DIFFINERY_ZONE` if |
| 115 | the Cloudflare zone isn't the domain's last two labels (for example |
| 116 | `example.co.uk`), and `MAIL_DOMAIN` to send email from another domain. The |
| 117 | token then needs Workers Routes Edit on that zone instead of `diffinery.com`. |
| 118 | |
| 119 | Email recovery uses Access's email one-time PIN, which works without any setup. |
| 120 | You can also add GitHub as a login method under Zero Trust → Settings → Authentication. |
| 121 | |
| 122 | ## Signing in and inviting people |
| 123 | |
| 124 | Everyone signs in with a passkey. There are no passwords. |
| 125 | |
| 126 | 1. **First sign-in for admins.** The emails in `BOOTSTRAP_ADMINS` (in |
| 127 | `apps/diffinery/wrangler.jsonc`) become site admins. Open |
| 128 | `https://diffinery.com/auth/access`, confirm your email with the one-time |
| 129 | code, and add a passkey on the page that follows. |
| 130 | 2. **Inviting someone.** An admin or org owner creates an invite from |
| 131 | *Settings → Site administration*, an organization's *People* tab, or |
| 132 | `diffinery invite friend@example.com --org codician`. Diffinery emails them |
| 133 | the link; they open it, pick a username and create a passkey, and they're in. |
| 134 | 3. **Lost passkey.** A site admin clicks *Sign-in link* next to the person in |
| 135 | *Settings → Site administration*; Diffinery emails them the one-time link, |
| 136 | which works for 24 hours. People can also recover through `/auth/access` by email. |
| 137 | 4. **More devices.** Add passkeys under *Settings → Passkeys and sessions*, or |
| 138 | keep one in a password manager that syncs. The same page signs out other |
| 139 | browsers. |
| 140 | 5. Repository and org roles (read, triage, write, maintain, admin) decide what |
| 141 | each person can do. |
| 142 | |
| 143 | Nobody needs a Cloudflare account. Git and the CLI use a personal token from |
| 144 | *Settings → Access tokens*. |
| 145 | |
| 146 | ```sh |
| 147 | git clone https://diffinery.com/codician/website.git # username: anything, password: your dfy_ token |
| 148 | diffinery login && diffinery auth setup-git # or let the CLI handle credentials |
| 149 | ``` |
| 150 | |
| 151 | |
| 152 | |
| 153 | Diffinery sends email through [Cloudflare Email Sending](https://developers.cloudflare.com/email-service/) |
| 154 | from `notifications@diffinery.com` (`MAIL_FROM` in `wrangler.jsonc`). Every |
| 155 | email has an HTML and a plain-text version, and they work like GitHub's: |
| 156 | |
| 157 | | Group | Emails | Can be turned off | |
| 158 | | --- | --- | --- | |
| 159 | | Security | New sign-in from an unknown browser or device, passkey added or removed, access token created or about to expire, agent added, push protection bypassed or a push blocked | No | |
| 160 | | Account | Invitations, sign-in links, welcome, account disabled or enabled, test email | No | |
| 161 | | Participating | Issues and pull requests you opened, commented on, reviewed, were assigned, asked to review, @mentioned in or subscribed to: new comments, reviews, pushes, merges, closes and reopens | Yes | |
| 162 | | Watching | Everything in repositories you watch with *All activity* | Yes | |
| 163 | | Actions | Failed workflow runs you started (all results if you ask) | Yes | |
| 164 | | Repositories | Repositories created or deleted in your organizations, repository access granted | Yes | |
| 165 | | Organizations | Organization role changes, accepted invitations, access requests (admins) | Yes | |
| 166 | |
| 167 | - **Settings.** *Settings → Notifications* turns each group on or off, on |
| 168 | Diffinery and by email, lists the repositories you watch and sends a test |
| 169 | email. Each repository has a *Watch* menu (participating, all activity, |
| 170 | ignore) and each issue and pull request a *Subscribe / Unsubscribe* button. |
| 171 | Your own comments and changes don't email you unless you ask. |
| 172 | - **Unsubscribing.** Emails carry `List-Unsubscribe` and one-click |
| 173 | unsubscribe headers (RFC 8058), so mail apps show an Unsubscribe button, and |
| 174 | the footer links mute the thread, stop watching the repository or turn off |
| 175 | that group. Links are signed, so they work without signing in. |
| 176 | - **Threading.** Emails about one issue or pull request share |
| 177 | `References`/`In-Reply-To` headers and a `[owner/repo]` subject, and carry |
| 178 | `X-Diffinery-Reason` (like `X-GitHub-Reason`) for filters. |
| 179 | - **Delivery.** Emails go through the `diffinery-mail` queue (retries with |
| 180 | backoff, then the `diffinery-mail-dlq` dead-letter queue) and are logged for |
| 181 | 90 days. Site admins see delivery status, the log and a preview of every |
| 182 | template under *Settings → Email*, and can send every template to |
| 183 | themselves. |
| 184 | - **API.** `GET/PUT/DELETE /api/v3/repos/{owner}/{repo}/subscription`, |
| 185 | `GET/PUT /api/v3/diffinery/notification-settings`, |
| 186 | `GET /api/v3/diffinery/emails`, `POST /api/v3/diffinery/emails/test`; for |
| 187 | site admins `GET /api/v3/diffinery/admin/email` and |
| 188 | `POST /api/v3/diffinery/admin/emails/samples`. |
| 189 | |
| 190 | **Setting it up.** Onboard the sending domain once in the Cloudflare dashboard |
| 191 | (*Compute → Email Service → Email Sending → Onboard Domain*); Cloudflare adds |
| 192 | the bounce MX, SPF, DKIM and DMARC records. Each deploy runs |
| 193 | `scripts/email-sending.sh`, which checks those records (with the *Email |
| 194 | Sending: Read* permission on the API token, or the public DNS records without |
| 195 | it) and warns if mail can't be delivered. In `wrangler dev`, Miniflare accepts |
| 196 | emails without sending them; `GET /api/v3/diffinery/emails` shows the full |
| 197 | message there. |
| 198 | |
| 199 | ## Agents |
| 200 | |
| 201 | Create an agent under *Settings → Agents* or with `diffinery agent create reviewer-bot --org codician`. |
| 202 | Then point any MCP client at it: |
| 203 | |
| 204 | ```json |
| 205 | { "mcpServers": { "diffinery": { "type": "http", "url": "https://diffinery.com/mcp", "headers": { "Authorization": "Bearer dfy_…" } } } } |
| 206 | ``` |
| 207 | |
| 208 | Tools include `get_file`, `list_tree`, `commit_files` (no git client needed), |
| 209 | `create_branch`, `create_pull_request` (with an `intent` for reviewers), |
| 210 | `review_pull_request`, `merge_pull_request` (policy-guarded), issues, |
| 211 | `list_workflow_runs` and `get_job_logs`, `rerun_workflow`, releases |
| 212 | (`create_release`, `upload_release_asset`, `generate_release_notes`), |
| 213 | `sandbox_create`/`sandbox_exec` for running builds and tests, and |
| 214 | `start_github_import`/`get_import`/`list_imports` to move repositories in |
| 215 | from GitHub (see below). |
| 216 | |
| 217 | Agent guardrails apply everywhere: git, REST, MCP and the web UI. |
| 218 | |
| 219 | - An agent's role is capped at write, and never exceeds its owner's role. |
| 220 | - Agents can't push to the default branch or protected branches, and can't |
| 221 | merge there. Turn this on per branch with `agentsMayPush` or `agentsMayMerge`. |
| 222 | - A pull request opened by an agent needs at least one human approval, and |
| 223 | approvals from agents don't count toward it. |
| 224 | - Agents can't change branch protection or read or set secrets. |
| 225 | - Agents can't change or delete environments or their branch policies, and |
| 226 | can't dismiss reviews. |
| 227 | - Commit statuses and check runs that agents report never satisfy a required |
| 228 | check. |
| 229 | |
| 230 | ## Workflows |
| 231 | |
| 232 | Put workflow files in `.github/workflows/` (or `.diffinery/workflows/`). |
| 233 | Supported triggers are `push`, `pull_request`, `pull_request_target`, |
| 234 | `merge_group`, `workflow_dispatch` (with inputs), `schedule` and `workflow_run` |
| 235 | (up to three levels deep, like GitHub), with branch, tag and path filters. |
| 236 | |
| 237 | Pull requests behave as on GitHub: |
| 238 | |
| 239 | - `pull_request` runs check out the test merge commit, `refs/pull/<n>/merge` |
| 240 | (the head merged into the current base), with `GITHUB_REF` set to that ref. |
| 241 | While a pull request conflicts with its base, its `pull_request` workflows |
| 242 | don't run. `refs/pull/<n>/head` and `refs/pull/<n>/merge` can be fetched but |
| 243 | not pushed. |
| 244 | - `pull_request_target` runs the base branch's own workflow on the base commit, |
| 245 | with the repository's secrets, so a pull request can't change what runs. |
| 246 | Check out the pull request's code yourself only if you trust it. |
| 247 | - Both report their checks on the pull request's head commit. `opened`, |
| 248 | `synchronize`, `reopened` and `closed` (with `pull_request.merged`) fire. |
| 249 | - Merge queue: turn on **Require a merge queue** in a branch protection rule |
| 250 | (or `"mergeQueue": {"method": "squash"}` through the API). "Merge when ready" |
| 251 | then adds a pull request to the queue. Each entry gets a merge group branch, |
| 252 | `gh-readonly-queue/<base>/pr-<n>-<head sha>`, holding the base, every pull |
| 253 | request ahead of it and its own changes, and `merge_group` (and `push`) |
| 254 | workflows run on it. Once its required checks pass and the entries ahead have |
| 255 | merged, the base fast-forwards to exactly the tested commit. A group that |
| 256 | fails, conflicts or exceeds the check timeout leaves the queue with a comment |
| 257 | saying why, and the groups behind it are rebuilt. The queue is at |
| 258 | `/<owner>/<repo>/queue/<branch>`; the API is |
| 259 | `PUT`/`DELETE /repos/{owner}/{repo}/pulls/{n}/merge-queue` and |
| 260 | `GET /repos/{owner}/{repo}/merge-queue/{branch}`, `diffinery pr merge` queues |
| 261 | automatically, and the MCP `merge_pull_request` tool does too. |
| 262 | |
| 263 | Supported workflow features: |
| 264 | |
| 265 | - `needs` and job outputs, `if` with `success()`, `failure()`, `always()` and |
| 266 | `cancelled()`, `strategy.matrix` with include and exclude, `fail-fast` and |
| 267 | `max-parallel`, `concurrency` with `cancel-in-progress` (for runs and for |
| 268 | jobs), and `timeout-minutes`. |
| 269 | - A job starts as soon as the jobs it `needs` have finished. It doesn't wait |
| 270 | for unrelated jobs. |
| 271 | - As on GitHub, a job is skipped when a job above it in the `needs` chain |
| 272 | failed, was cancelled or was skipped, unless its `if:` uses `always()`, |
| 273 | `failure()` or `cancelled()`. `!cancelled() && !failure()` runs it after a |
| 274 | skip. `needs.<job>.result` is `skipped` for a skipped job. |
| 275 | - `runs-on`, `name`, `environment`, `timeout-minutes`, `continue-on-error` |
| 276 | and job `concurrency` can use the `matrix` and `strategy` contexts, and are |
| 277 | worked out for each matrix job, e.g. `runs-on: ${{ matrix.os }}`. |
| 278 | - A job-level `concurrency` group lets one job run at a time across the |
| 279 | repository. Others wait as pending; a newer pending job replaces an older |
| 280 | one. |
| 281 | - Job `continue-on-error`: a failed job doesn't fail the run, doesn't cancel |
| 282 | the rest of its matrix, and counts as a success for the jobs that need it. |
| 283 | - `continue-on-error`, `defaults.run`, `env` at every level, `$GITHUB_OUTPUT`, |
| 284 | `$GITHUB_ENV`, `$GITHUB_PATH`, `$GITHUB_STEP_SUMMARY`, and masking of |
| 285 | secrets and `::add-mask::` values. |
| 286 | - `secrets.*`, `vars.*`, and `GITHUB_TOKEN`, a per-job token that follows |
| 287 | GitHub's `permissions:` model: workflow- or job-level `read-all`, |
| 288 | `write-all`, `{}` or per-scope `contents`, `issues`, `pull-requests`, |
| 289 | `statuses`, `checks`, `deployments`, `actions`, `packages`, `pages` and so |
| 290 | on. Without `permissions:` a job gets the repository's default, which is |
| 291 | GitHub's restricted one (`contents` and `packages` read) for new |
| 292 | repositories, and write for ones created before the setting existed. An |
| 293 | admin changes it, and whether |
| 294 | workflows may create and approve pull requests (off by default), with |
| 295 | `PUT /repos/{owner}/{repo}/actions/permissions/workflow` |
| 296 | (`default_workflow_permissions`, `can_approve_pull_request_reviews`). Even |
| 297 | when allowed, a workflow's approval doesn't count toward required approvals. |
| 298 | A reusable workflow never gets more than its caller, and a pull request from |
| 299 | another repository only reads. Each API endpoint needs its scope (an issue |
| 300 | write without `issues: write` is a 403, like GitHub's "Resource not |
| 301 | accessible by integration"). The token never administers the repository, |
| 302 | reads secrets or creates or changes workflow files (GitHub's `workflows` |
| 303 | permission), and branch protection applies to it. A run whose code came from |
| 304 | an agent (the agent pushed it or the branch, or opened the pull request, and |
| 305 | re-runs of such runs) is held to the agent rules: its token can't push or |
| 306 | merge to the default or protected branches. `permissions: id-token: write` |
| 307 | enables OIDC (issuer `https://diffinery.com`). |
| 308 | - `runs-on: ubuntu-latest` (and any `ubuntu-*` or `linux` label) runs in a |
| 309 | Cloudflare Container. `runs-on: [self-hosted, …]` jobs wait for a runner you |
| 310 | start with `diffinery runner -R owner/repo --labels gpu,macos`. Use that for |
| 311 | macOS, GPUs or private networks. |
| 312 | - Reusable workflows: `jobs.<id>.uses: ./.github/workflows/x.yml` from the same |
| 313 | commit, or `owner/repo/.github/workflows/x.yml@ref` from another repository |
| 314 | on this instance (public ones, or private ones with the same |
| 315 | owner). `with`, `secrets` (a mapping or `inherit`), `on.workflow_call` |
| 316 | outputs and nesting up to four levels deep all work. A matrix on the calling |
| 317 | job isn't supported yet. |
| 318 | - Docker, on hosted runners and on self-hosted runners with Docker installed: |
| 319 | a job `container:` (steps and JavaScript actions run inside it), `services:` |
| 320 | (reachable by name from the job container, or on `localhost` through their |
| 321 | mapped ports, with health checks awaited), Docker container actions |
| 322 | (`docker://image` steps and actions with `runs.using: docker`, built from |
| 323 | their Dockerfile), and plain `docker build` / `docker run` / `docker compose` |
| 324 | in steps. Hosted runners start the Docker daemon the first time a job uses |
| 325 | it and pull Docker Hub images through `mirror.gcr.io`. |
| 326 | - A container registry at `diffinery.com` in place of ghcr.io: sign in with |
| 327 | the job's `GITHUB_TOKEN` (`permissions: packages: write`) or a personal |
| 328 | token, then push and pull `diffinery.com/<owner>/<name>`. Images are |
| 329 | private by default and can be linked to a repository. See |
| 330 | [Container registry](https://diffinery.com/docs/packages/container-registry). |
| 331 | - Actions from GitHub run unmodified, including `actions/github-script`, |
| 332 | `setup-java`, `setup-dotnet` and the rest. Jobs get the same runtime |
| 333 | services GitHub provides (`ACTIONS_RUNTIME_TOKEN`, `ACTIONS_RESULTS_URL`, |
| 334 | `ACTIONS_CACHE_URL`), scoped to the job and expiring with it, so |
| 335 | `actions/cache` (v3 and v4), `actions/upload-artifact` and |
| 336 | `actions/download-artifact` (v4: `pattern`, `merge-multiple`, `run-id` with |
| 337 | `github-token`) and `docker/build-push-action` with `cache-to: type=gha` store |
| 338 | caches and artifacts in Diffinery. `GITHUB_EVENT_PATH` holds a complete event |
| 339 | payload and `github.run_id` is numeric. |
| 340 | Caches and artifacts are listed and deleted through the REST API |
| 341 | (`/actions/caches`, `/actions/artifacts`). Each repository keeps up to 10 GB |
| 342 | of caches (the least recently used are evicted) and 10 GB of artifacts |
| 343 | (`ARTIFACT_QUOTA_BYTES` changes it; uploads beyond it are refused). Uploads |
| 344 | in progress count toward both, and a job's unfinished uploads are removed |
| 345 | when it ends. As on GitHub, a cache is saved for the ref the job runs on and |
| 346 | restored from that ref, then the pull request's base branch, then the |
| 347 | default branch, so branches and pull requests never see each other's |
| 348 | caches. Artifacts keep their `retention-days` (1 to 90 days, 90 by default); |
| 349 | an expired artifact's content is deleted, and the REST API still lists it |
| 350 | with `expired: true` (downloads get 410) for another 90 days. |
| 351 | - `actions/checkout`, `cache`, `upload-artifact`, `download-artifact`, |
| 352 | `setup-node`, `setup-python` and `setup-go` run as built-ins for speed. They |
| 353 | share storage and formats with the real actions (a cache saved by one |
| 354 | restores with the other) and support their inputs, including checkout's |
| 355 | `lfs`, `submodules`, `sparse-checkout`, `filter`, `fetch-tags`, `ref` and |
| 356 | `persist-credentials: false`, and the `cache:` input of the setup actions. |
| 357 | A built-in hands over to the real action when it sees an input or ref it |
| 358 | doesn't mirror (a commit SHA, a branch, a newer major). To always run the |
| 359 | real action, set `DIFFINERY_BUILTINS: false` in a workflow, job or step |
| 360 | `env:`, or start a self-hosted runner with `DIFFINERY_DISABLE_BUILTINS=1`. |
| 361 | - Hosted runners look like GitHub's `ubuntu-24.04` image to actions: |
| 362 | `ImageOS=ubuntu24`, `RUNNER_TOOL_CACHE=/opt/hostedtoolcache`, and `git`, |
| 363 | `git-lfs`, `jq`, `zip`, `zstd`, `curl`, `python3`/`pip`, `build-essential`, |
| 364 | `docker` and `gh` installed. `ACTIONS_STEP_DEBUG` and `ACTIONS_RUNNER_DEBUG` |
| 365 | (as secrets or variables) turn on debug logging. |
| 366 | |
| 367 | **Not supported yet:** Windows and macOS hosted runners, the GraphQL API |
| 368 | (`github.graphql` in github-script), and `deepen-not` / partial clone filters |
| 369 | on the server (checkout's `filter` falls back to a full fetch). One cache or |
| 370 | artifact upload request goes through the Worker, so it is limited to the |
| 371 | Worker's request size (100 MB on most plans); the built-ins and the official |
| 372 | actions upload large files in blocks. A job that uses an unsupported feature fails |
| 373 | with a clear message instead of misbehaving. |
| 374 | |
| 375 | ## REST API |
| 376 | |
| 377 | The API lives at `/api/v3`, with the same paths, JSON shapes and errors as |
| 378 | GitHub's REST API. Point a client at it and it works: |
| 379 | |
| 380 | ```js |
| 381 | const octokit = new Octokit({ auth: token, baseUrl: "https://diffinery.example.com/api/v3" }); |
| 382 | ``` |
| 383 | |
| 384 | Responses carry `node_id`, `url`, `html_url`, full user objects and label |
| 385 | objects. Lists take `per_page` and `page` and send a `Link` header, so |
| 386 | `octokit.paginate` works. Errors have `message` and `documentation_url`. |
| 387 | Responses also keep a few Diffinery fields (such as `merge_blockers` on pull |
| 388 | requests) for the CLI, MCP and web UI. |
| 389 | |
| 390 | What works: |
| 391 | |
| 392 | - **Repositories:** get, create, update, delete, topics, languages, |
| 393 | collaborators, branches, tags, readme, contents (GET, and PUT and DELETE |
| 394 | with `sha`; a stale `sha` returns 409), compare, commits, `merges`, |
| 395 | `dispatches`, and `zipball`/`tarball` archives. `private: false` makes a |
| 396 | repository public. |
| 397 | - **Archives:** `zipball` and `tarball` are built on demand and cached in R2. |
| 398 | The cache can be evicted at any time; the archive is then built again. An |
| 399 | archive holds at most 24 MB of files and 100,000 tree entries. A zip holds |
| 400 | at most 65,535 files; use the tarball for more. Each repository can build |
| 401 | 20 new archives per hour. Clone bigger repositories with git. |
| 402 | - **Git data:** blobs, trees (with `base_tree` and `recursive`), commits, |
| 403 | annotated tags, `git/ref`, `git/refs` and `git/matching-refs`. |
| 404 | - **Releases:** list, get by id or tag, latest, create (the tag is created |
| 405 | if missing), update, delete, `generate-notes` (honours |
| 406 | `.github/release.yml`), `make_latest`, and drafts (visible to writers only). |
| 407 | Upload assets to the release's `upload_url`. Download them from |
| 408 | `browser_download_url`, or from the asset URL with |
| 409 | `Accept: application/octet-stream`. Bytes live in R2. An asset can be up to |
| 410 | 100 MB, the largest request body Workers accept. |
| 411 | - **Commit statuses:** create, list and the combined status. A status counts |
| 412 | as a check, so a branch protection's required checks can name a status |
| 413 | context. |
| 414 | - **Required checks:** when Diffinery Actions reports a check, it alone |
| 415 | decides that check. To have a required check decided by someone else, bind |
| 416 | it in branch protection, for example |
| 417 | `"requiredCheckSources": { "ci/external": "ci-bot" }`. A required check |
| 418 | fails if any counted report of it fails. Only the reporter that created a |
| 419 | check run can update it, and Actions check runs can't be changed over REST. |
| 420 | - **Checks:** check runs (create, update, get, annotations) and check suites. |
| 421 | - **Issues and pull requests:** labels, assignees, lock and unlock, |
| 422 | comments (by issue and repository-wide), reactions (issues, comments, review |
| 423 | comments and releases), events and the timeline, reviews, review comments |
| 424 | and replies, review dismissal, requested reviewers, `update-branch`, merge, |
| 425 | `commits/{sha}/pulls`, and list filters (`head`, `base`, `sort`, |
| 426 | `direction`, `labels`, `assignee`, `since`). Pull requests report |
| 427 | `mergeable`, `mergeable_state` and `rebaseable`. Comment, review, check |
| 428 | run, user and repository ids are numbers, as on GitHub. Locked |
| 429 | conversations refuse comments, review comments and reactions from people |
| 430 | without triage access. Dismissing a review takes maintain access, and the |
| 431 | pull request's author can't do it. |
| 432 | - **Drafts and merging:** `PATCH /pulls/{n}` with `"draft": false` marks a |
| 433 | draft ready for review (`"draft": true` turns it back; GitHub only offers |
| 434 | this through GraphQL). The web page has a "Ready for review" button and the |
| 435 | CLI has `diffinery pr ready`. A merge commit is refused on a branch that |
| 436 | requires linear history. Admins can merge past unmet branch rules only by |
| 437 | sending `"bypass_rules": true` (the CLI's `pr merge --admin`); the bypass is |
| 438 | written to the audit log. When several rules match a branch, the rule that |
| 439 | names it exactly wins, then the oldest pattern. |
| 440 | - **Who can write:** anonymous callers can read public repositories but can't |
| 441 | create or change anything. A `read`-scoped token only reads; creating |
| 442 | organizations, invitations, agents and tokens takes an `admin`-scoped token |
| 443 | or a browser session. Token scopes must be `read`, `write` or `admin`. |
| 444 | Agents can't add, change or delete secrets or variables, whatever their |
| 445 | role. |
| 446 | - **Archived repositories** are read-only through the API, for everyone. |
| 447 | - **Tree entry names:** every write refuses `.`, `..`, `.git` (in any case, |
| 448 | and spellings like `.git.` or `git~1`), and names with `\` or NUL. |
| 449 | - **Deployments and environments:** deployments, deployment statuses, |
| 450 | environments, deployment branch policies, and environment secrets and |
| 451 | variables. A deployment without `required_contexts` needs every context on |
| 452 | the commit to have passed, as on GitHub; send `[]` to skip that. An |
| 453 | environment with no branch policy gets "protected branches only" when its |
| 454 | first secret is added. Environment secrets never go to `refs/pull/*` runs. |
| 455 | Runs started by agents only get them when the environment allows protected |
| 456 | branches only. Tags pushed by agents never match an environment's tag |
| 457 | patterns. |
| 458 | - **Actions:** runs, run attempts, jobs, logs (a zip per run), artifacts, |
| 459 | workflows, secrets and variables. |
| 460 | - **Search:** `/search/issues` (`repo:`, `is:`, `label:`, `author:`, |
| 461 | `assignee:`, `no:`, `head:`, `base:`, `in:`, date ranges and `sort:`) and |
| 462 | `/search/repositories` (`user:`, `org:`, `topic:`, `is:public`, |
| 463 | `archived:`). Search only covers repositories you can read. A query |
| 464 | can be up to 256 characters, and one search looks at up to 50 repositories. |
| 465 | - `/rate_limit`. Diffinery doesn't rate-limit the API, so it reports a |
| 466 | generous budget. |
| 467 | |
| 468 | Workflows can react to these too. A published release triggers `on: release`, |
| 469 | a `POST /dispatches` triggers `on: repository_dispatch`, and webhooks get |
| 470 | `release`, `status`, `deployment`, `deployment_status` and |
| 471 | `workflow_dispatch` events, and `edited`, `ready_for_review` and |
| 472 | `converted_to_draft` actions on issues and pull requests. A delivery that |
| 473 | fails with a network error, 408, 429 or 5xx is retried up to five times over |
| 474 | about five hours. A job with |
| 475 | `environment:` creates a deployment, gets that environment's secrets and |
| 476 | variables, and is refused if the environment's branch policy doesn't allow |
| 477 | the ref. As on GitHub, events caused by `GITHUB_TOKEN` don't start new |
| 478 | workflow runs, except `repository_dispatch`. |
| 479 | |
| 480 | **Not supported yet:** GraphQL (`/api/graphql`), setting secrets with a |
| 481 | sealed-box `encrypted_value` (send `value` instead), environment protection |
| 482 | rules (required reviewers, wait timers), forks, team reviewers, and code and |
| 483 | commit search. Search qualifiers for stars, forks and languages are accepted |
| 484 | but ignored. |
| 485 | |
| 486 | ## Moving from GitHub |
| 487 | |
| 488 | Diffinery imports a GitHub repository — or a whole organization — on the server: |
| 489 | it copies the Git history in a container and the rest through GitHub's REST API, |
| 490 | so you don't clone or push anything yourself and large repositories don't run |
| 491 | through your laptop. |
| 492 | |
| 493 | Start an import three ways: |
| 494 | |
| 495 | - **Web** — *Settings → Imports → New*, or the *Import from GitHub* link on *New |
| 496 | repository*. Paste `owner/name` (or a GitHub URL), choose where it lands and |
| 497 | whether it stays public or private, and watch each step. For an organization, |
| 498 | pick the repositories from the list. |
| 499 | - **CLI** — `diffinery migrate github octo/app` imports one repository and |
| 500 | follows it to completion; `diffinery migrate github-org octo --to my-org` |
| 501 | imports an organization. Both take `--json`. `--local` runs the older |
| 502 | client-side path instead (a mirror push plus a best-effort recreation of |
| 503 | issues and open pull requests). |
| 504 | - **MCP / agents** — `start_github_import`, then `get_import` to poll and |
| 505 | `list_imports` to review. An agent can import only into its owner's account or |
| 506 | an organization the owner administers. |
| 507 | |
| 508 | A private repository, Git LFS, Actions variables and branch protection need a |
| 509 | GitHub token (`GH_TOKEN`/`GITHUB_TOKEN`, or the token field in the form) with |
| 510 | `repo` access. For GitHub Enterprise you must name the API base explicitly with |
| 511 | `GITHUB_API_URL` (or the API URL field in the form): when a token is supplied for |
| 512 | a non-github.com source, Diffinery requires a matching `github_api_url` on the |
| 513 | same host and attaches the token only there, so a host taken from the source can |
| 514 | never receive it. Server-side fetches only ever reach that one host, every |
| 515 | response body is size-bounded, redirects are not followed across hosts, and the |
| 516 | token goes to GitHub alone (never logged, never returned). |
| 517 | |
| 518 | **Limits.** Each account (and each organization) may run **one** organization |
| 519 | import and **three** single-repository imports at a time. A repository larger |
| 520 | than `IMPORT_MAX_REPO_KB` (default 10 GB) is refused up front. The Git transfer |
| 521 | has a **6-hour** ceiling; an import that is still transferring after that fails |
| 522 | cleanly. Draft imports left unstarted for seven days are cancelled and their |
| 523 | stored token is erased. |
| 524 | |
| 525 | **Agents.** An agent may import only into its owner's account, or an organization |
| 526 | the owner administers, and only when the owner holds at least **write** there. |
| 527 | Agents never read or set secrets and never manage **variables**: the repository, |
| 528 | organization and environment **variable** steps are skipped for an |
| 529 | agent-started import (and re-checked against a human owner when they do run). An |
| 530 | agent-started import also never bypasses push protection — if the history is not |
| 531 | clean, it stops. |
| 532 | |
| 533 | What comes across, keeping original numbers and attribution: |
| 534 | |
| 535 | - **Git** — every branch and tag, and LFS objects. Pull-request head commits are |
| 536 | kept so merged and open PRs still point at real commits. Merge-queue refs are |
| 537 | skipped. |
| 538 | - **Issues and pull requests** — with their **original numbers** (gaps and all), |
| 539 | bodies, comments, reviews and review threads, labels, milestones, assignees, |
| 540 | merged/closed state and who merged. |
| 541 | - **Labels, milestones, branch protection**, and **Actions variables** (repository |
| 542 | and organization) and **deployment environments** (names, branch policies and |
| 543 | variables). |
| 544 | - **Releases** and their assets, where the server's releases feature is enabled. |
| 545 | |
| 546 | What does **not** come across: |
| 547 | |
| 548 | - **Secret values.** GitHub never discloses them, and Diffinery never tries to. |
| 549 | Repository and environment secret **names** are listed so you can re-enter the |
| 550 | values by hand (`diffinery secret set <NAME> -R owner/repo`), which an admin |
| 551 | does — never an agent. |
| 552 | - **People.** No accounts are created. A GitHub user is attributed to a Diffinery |
| 553 | account only through a verified link, matched by GitHub's **numeric id** so a |
| 554 | rename can't mis-map and keyed by the full API base so a github.com link and an |
| 555 | Enterprise link stay separate. Everyone else is imported as a **mannequin**: a |
| 556 | non-account placeholder (`github:<login>`, linking to the GitHub profile) that |
| 557 | carries the history but can't sign in, be assigned work or cast a vote. Where a |
| 558 | mannequin authors a squashed merge, the commit uses GitHub's `noreply` address. |
| 559 | |
| 560 | Link a GitHub account under *Settings → Linked accounts → GitHub* one of two ways: |
| 561 | |
| 562 | - **Gist proof (no token).** Diffinery shows a one-time code; you put it in a |
| 563 | **public gist** and confirm your login. Diffinery reads your public profile and |
| 564 | gists **unauthenticated** on the API base you named and links the account only |
| 565 | when it finds the code. Nothing but your public data is read. |
| 566 | - **OAuth (optional).** When `GITHUB_OAUTH_CLIENT_ID` and |
| 567 | `GITHUB_OAUTH_CLIENT_SECRET` are configured, a *Connect GitHub* button runs the |
| 568 | OAuth flow with state bound to your account so a callback can't be replayed. |
| 569 | |
| 570 | A site admin can clear any link (audited). Imported reviews are kept as history |
| 571 | but **never count as approvals**, even when their author is a linked account — a |
| 572 | pull request still needs a fresh review on Diffinery to be approved. |
| 573 | |
| 574 | Old history that contains a credential is imported with an audited push-protection |
| 575 | bypass (the history is already public on GitHub); choose *stop* to fail instead. |
| 576 | |
| 577 | **Retrying.** Only the person who started an import may retry it, and a retry |
| 578 | needs the GitHub token again (it is never stored past the run). While an import |
| 579 | is in progress the target repository is **locked** against other writes, and a |
| 580 | retry refuses to touch a repository that already holds content the import did not |
| 581 | write — it will not adopt or overwrite an existing repository. An agent may not |
| 582 | retry once the repository has been created. |
| 583 | |
| 584 | ## Developing |
| 585 | |
| 586 | ```sh |
| 587 | pnpm install |
| 588 | pnpm -r build && pnpm -r typecheck && pnpm -r test |
| 589 | pnpm --filter @diffinery/app db:migrate:local |
| 590 | pnpm dev # http://localhost:8787, signed in as dev@diffinery.local |
| 591 | pnpm --filter @diffinery/app e2e # wrangler dev + real git + the runner agent + Chromium |
| 592 | ``` |
| 593 | |
| 594 | The end-to-end test does the following: |
| 595 | |
| 596 | 1. Pushes a repository with real `git`. |
| 597 | 2. Runs its CI through the actual runner agent. |
| 598 | 3. Checks secret scanning and branch protection. |
| 599 | 4. Has an agent open a pull request that needs a human approval and a green |
| 600 | check, then merges it. |
| 601 | 5. Runs pull request workflows on test merge commits, `pull_request_target` |
| 602 | from the base branch, and a merge queue with `merge_group` runs, including a |
| 603 | group that fails and leaves the queue. |
| 604 | 6. Clones shallow, then deepens the clone. |
| 605 | 7. Drives the MCP server. |
| 606 | 8. In Chromium with a virtual authenticator, joins from an invite with a new |
| 607 | passkey, adds and removes passkeys, signs out and in, and uses an admin |
| 608 | sign-in link. |
| 609 | 9. Drives the REST API with the real `@octokit/rest` client: contents, git |
| 610 | data, statuses as required checks, releases with asset uploads and |
| 611 | downloads, issues, reactions, pagination, search and deployments. |
| 612 | |
| 613 | Team preferences for product and design decisions are in |
| 614 | [docs/PREFERENCES.md](docs/PREFERENCES.md). |