diffinery/README.md
| 1 | # Diffinery |
| 2 | |
| 3 | A git forge for teams of people and AI agents, running entirely on Cloudflare. |
| 4 | Git hosting, pull requests and reviews, issues, GitHub-Actions-compatible CI, |
| 5 | an MCP server for agents, and sandboxes, with no servers to run. |
| 6 | |
| 7 | - **Drop-in for GitHub workflows.** `.github/workflows/*.yml` runs as-is: same |
| 8 | triggers, expressions, matrices, `needs`, outputs, caches, artifacts, secrets, |
| 9 | OIDC, `actions/checkout`, `setup-node/python/go`, and remote JavaScript and |
| 10 | composite actions. |
| 11 | - **Built for agents.** Agents are first-class accounts owned by a person or |
| 12 | team, with their own tokens and limits. They can't push or merge to protected |
| 13 | branches, and their pull requests always need a human approval. Every commit |
| 14 | they make is stamped with who they act for. They work through MCP |
| 15 | (`/mcp`) or the REST API, and can start sandboxes to build and test. |
| 16 | - **Secure by default.** People sign in with passkeys, with no passwords to |
| 17 | leak or phish. Invites, roles and branch protection decide what they can do. Pushes are scanned for |
| 18 | leaked credentials and blocked if they contain any; nothing from a blocked |
| 19 | push is kept. A repository admin (a person, never an agent) can push anyway |
| 20 | with `git push -o allow-secrets=<reason>`, which is audited. A file can opt |
| 21 | out with a `diffinery:allow-secret` comment, for test fixtures. Secrets use |
| 22 | envelope encryption, and every sensitive action goes to an append-only audit log. |
| 23 | |
| 24 | ## How it fits on Cloudflare |
| 25 | |
| 26 | | Piece | Cloudflare product | |
| 27 | |---|---| |
| 28 | | HTTP, git smart HTTP, REST API, MCP, web UI | Workers (Hono) | |
| 29 | | One repository: refs, objects, PRs, issues, reviews, checks, runs | Durable Object with SQLite (`RepoDO`), one per repository | |
| 30 | | Large objects, Git LFS, container images, logs, caches, artifacts, event archive | R2 | |
| 31 | | Accounts, orgs, tokens, invites, notifications, audit log | D1, read from nearby replicas through the Sessions API (each browser keeps a bookmark so it reads its own writes) | |
| 32 | | Webhooks, notifications, audit fan-out | Queues (with a dead-letter queue) | |
| 33 | | Workflow runs (durable, survive restarts, 6 h job timeout) | Workflows | |
| 34 | | Job runners and agent sandboxes | Containers (Ubuntu 24.04 image in `containers/runner`) | |
| 35 | | Scheduled workflows (`on: schedule`) | Cron Triggers | |
| 36 | | Live run logs and PR updates | Durable Object WebSockets (hibernation) | |
| 37 | | Sign-in | Passkeys (WebAuthn) and sessions in D1. Cloudflare Access guards only `/auth/access`, the email recovery path | |
| 38 | |
| 39 | ``` |
| 40 | apps/diffinery the Worker (src/index.ts), D1 migrations, wrangler config, e2e test |
| 41 | apps/cli the `diffinery` command-line client |
| 42 | containers/runner runner agent + sandbox server, and the container image |
| 43 | packages/git-core git objects, packfiles, deltas, diff/merge (pure TypeScript) |
| 44 | packages/workflow-yaml GitHub Actions workflow parser, expressions, matrix, cron |
| 45 | packages/shared types shared by the Worker and the runner |
| 46 | scripts bootstrap.sh, ensure-secrets.sh, render-config.mjs |
| 47 | ``` |
| 48 | |
| 49 | ## Deploying |
| 50 | |
| 51 | Diffinery deploys itself: this repository lives at |
| 52 | `diffinery.com/codician-team/diffinery`, and every push to `main` there runs |
| 53 | `.github/workflows/deploy.yml` on Diffinery's hosted runners, followed by the |
| 54 | live test (`.github/workflows/live-test.yml`). GitHub stays the upstream for |
| 55 | now: `.github/workflows/sync-to-diffinery.yml` copies every branch and tag, and |
| 56 | the secrets and variables below, to Diffinery on each push, then waits for the |
| 57 | Actions runs there and fails if any of them fail. Running Deploy by hand on |
| 58 | GitHub still deploys, as a fallback. |
| 59 | |
| 60 | The workflow needs two repository secrets: |
| 61 | |
| 62 | - `CLOUDFLARE_API_TOKEN`: account permissions Workers Scripts, D1, R2, |
| 63 | Queues, Workflows, Containers and Access: Apps and Policies (all Edit), and |
| 64 | zone permission Workers Routes Edit on `diffinery.com`. |
| 65 | - `CLOUDFLARE_ACCOUNT_ID` |
| 66 | |
| 67 | On each deploy, `scripts/bootstrap.sh` creates anything that is missing: the D1 |
| 68 | database, R2 bucket, queues and one Access application that covers only |
| 69 | `/auth/access` (email-based account recovery). It then applies D1 migrations. `wrangler deploy` |
| 70 | builds and pushes the runner image. On the first deploy, `ensure-secrets.sh` |
| 71 | generates `MASTER_KEY` and the OIDC signing key inside the job, so neither |
| 72 | value is ever printed. |
| 73 | |
| 74 | If the token can't read your Zero Trust organization, set a repository |
| 75 | **variable** `ACCESS_TEAM_DOMAIN` (for example `codician.cloudflareaccess.com`). |
| 76 | |
| 77 | Email recovery uses Access's email one-time PIN, which works without any setup. |
| 78 | You can also add GitHub as a login method under Zero Trust → Settings → Authentication. |
| 79 | |
| 80 | ## Signing in and inviting people |
| 81 | |
| 82 | Everyone signs in with a passkey. There are no passwords. |
| 83 | |
| 84 | 1. **First sign-in for admins.** The emails in `BOOTSTRAP_ADMINS` (in |
| 85 | `apps/diffinery/wrangler.jsonc`) become site admins. Open |
| 86 | `https://diffinery.com/auth/access`, confirm your email with the one-time |
| 87 | code, and add a passkey on the page that follows. |
| 88 | 2. **Inviting someone.** An admin or org owner creates an invite from |
| 89 | *Settings → Site administration*, an organization's *People* tab, or |
| 90 | `diffinery invite friend@example.com --org codician`. Diffinery emails them |
| 91 | the link; they open it, pick a username and create a passkey, and they're in. |
| 92 | 3. **Lost passkey.** A site admin clicks *Sign-in link* next to the person in |
| 93 | *Settings → Site administration*; Diffinery emails them the one-time link, |
| 94 | which works for 24 hours. People can also recover through `/auth/access` by email. |
| 95 | 4. **More devices.** Add passkeys under *Settings → Passkeys and sessions*, or |
| 96 | keep one in a password manager that syncs. The same page signs out other |
| 97 | browsers. |
| 98 | 5. Repository and org roles (read, triage, write, maintain, admin) decide what |
| 99 | each person can do. |
| 100 | |
| 101 | Nobody needs a Cloudflare account. Git and the CLI use a personal token from |
| 102 | *Settings → Tokens*. |
| 103 | |
| 104 | ```sh |
| 105 | git clone https://diffinery.com/codician/website.git # username: anything, password: your dfy_ token |
| 106 | diffinery login && diffinery auth setup-git # or let the CLI handle credentials |
| 107 | ``` |
| 108 | |
| 109 | |
| 110 | |
| 111 | Diffinery sends email through [Cloudflare Email Sending](https://developers.cloudflare.com/email-service/) |
| 112 | from `notifications@diffinery.com` (`MAIL_FROM` in `wrangler.jsonc`). Every |
| 113 | email has an HTML and a plain-text version, and they work like GitHub's: |
| 114 | |
| 115 | | Group | Emails | Can be turned off | |
| 116 | | --- | --- | --- | |
| 117 | | Security | New sign-in from an unknown browser or device, passkey added or removed, access token created or about to expire, agent added, push protection bypassed or a push blocked | No | |
| 118 | | Account | Invitations, sign-in links, welcome, account disabled or enabled, test email | No | |
| 119 | | Participating | Issues and pull requests you opened, commented on, reviewed, were assigned, asked to review, @mentioned in or subscribed to: new comments, reviews, pushes, merges, closes and reopens | Yes | |
| 120 | | Watching | Everything in repositories you watch with *All activity* | Yes | |
| 121 | | Actions | Failed workflow runs you started (all results if you ask) | Yes | |
| 122 | | Repositories | Repositories created or deleted in your organizations, repository access granted | Yes | |
| 123 | | Organizations | Organization role changes, accepted invitations, access requests (admins) | Yes | |
| 124 | |
| 125 | - **Settings.** *Settings → Notifications* turns each group on or off, on |
| 126 | Diffinery and by email, lists the repositories you watch and sends a test |
| 127 | email. Each repository has a *Watch* menu (participating, all activity, |
| 128 | ignore) and each issue and pull request a *Subscribe / Unsubscribe* button. |
| 129 | Your own comments and changes don't email you unless you ask. |
| 130 | - **Unsubscribing.** Emails carry `List-Unsubscribe` and one-click |
| 131 | unsubscribe headers (RFC 8058), so mail apps show an Unsubscribe button, and |
| 132 | the footer links mute the thread, stop watching the repository or turn off |
| 133 | that group. Links are signed, so they work without signing in. |
| 134 | - **Threading.** Emails about one issue or pull request share |
| 135 | `References`/`In-Reply-To` headers and a `[owner/repo]` subject, and carry |
| 136 | `X-Diffinery-Reason` (like `X-GitHub-Reason`) for filters. |
| 137 | - **Delivery.** Emails go through the `diffinery-mail` queue (retries with |
| 138 | backoff, then the `diffinery-mail-dlq` dead-letter queue) and are logged for |
| 139 | 90 days. Site admins see delivery status, the log and a preview of every |
| 140 | template under *Settings → Email*, and can send every template to |
| 141 | themselves. |
| 142 | - **API.** `GET/PUT/DELETE /api/v3/repos/{owner}/{repo}/subscription`, |
| 143 | `GET/PUT /api/v3/diffinery/notification-settings`, |
| 144 | `GET /api/v3/diffinery/emails`, `POST /api/v3/diffinery/emails/test`; for |
| 145 | site admins `GET /api/v3/diffinery/admin/email` and |
| 146 | `POST /api/v3/diffinery/admin/emails/samples`. |
| 147 | |
| 148 | **Setting it up.** Onboard the sending domain once in the Cloudflare dashboard |
| 149 | (*Compute → Email Service → Email Sending → Onboard Domain*); Cloudflare adds |
| 150 | the bounce MX, SPF, DKIM and DMARC records. Each deploy runs |
| 151 | `scripts/email-sending.sh`, which checks those records (with the *Email |
| 152 | Sending: Read* permission on the API token, or the public DNS records without |
| 153 | it) and warns if mail can't be delivered. In `wrangler dev`, Miniflare accepts |
| 154 | emails without sending them; `GET /api/v3/diffinery/emails` shows the full |
| 155 | message there. |
| 156 | |
| 157 | ## Agents |
| 158 | |
| 159 | Create an agent under *Settings → Agents* or with `diffinery agent create reviewer-bot --org codician`. |
| 160 | Then point any MCP client at it: |
| 161 | |
| 162 | ```json |
| 163 | { "mcpServers": { "diffinery": { "type": "http", "url": "https://diffinery.com/mcp", "headers": { "Authorization": "Bearer dfy_…" } } } } |
| 164 | ``` |
| 165 | |
| 166 | Tools include `get_file`, `list_tree`, `commit_files` (no git client needed), |
| 167 | `create_branch`, `create_pull_request` (with an `intent` for reviewers), |
| 168 | `review_pull_request`, `merge_pull_request` (policy-guarded), issues, |
| 169 | `list_workflow_runs` and `get_job_logs`, `rerun_workflow`, and |
| 170 | `sandbox_create`/`sandbox_exec` for running builds and tests. |
| 171 | |
| 172 | Agent guardrails apply everywhere: git, REST, MCP and the web UI. |
| 173 | |
| 174 | - An agent's role is capped at write, and never exceeds its owner's role. |
| 175 | - Agents can't push to the default branch or protected branches, and can't |
| 176 | merge there. Turn this on per branch with `agentsMayPush` or `agentsMayMerge`. |
| 177 | - A pull request opened by an agent needs at least one human approval, and |
| 178 | approvals from agents don't count toward it. |
| 179 | - Agents can't change branch protection or read or set secrets. |
| 180 | |
| 181 | ## Workflows |
| 182 | |
| 183 | Put workflow files in `.github/workflows/` (or `.diffinery/workflows/`). |
| 184 | Supported triggers are `push`, `pull_request`, `pull_request_target`, |
| 185 | `merge_group`, `workflow_dispatch` (with inputs), `schedule` and `workflow_run` |
| 186 | (up to three levels deep, like GitHub), with branch, tag and path filters. |
| 187 | |
| 188 | Pull requests behave as on GitHub: |
| 189 | |
| 190 | - `pull_request` runs check out the test merge commit, `refs/pull/<n>/merge` |
| 191 | (the head merged into the current base), with `GITHUB_REF` set to that ref. |
| 192 | While a pull request conflicts with its base, its `pull_request` workflows |
| 193 | don't run. `refs/pull/<n>/head` and `refs/pull/<n>/merge` can be fetched but |
| 194 | not pushed. |
| 195 | - `pull_request_target` runs the base branch's own workflow on the base commit, |
| 196 | with the repository's secrets, so a pull request can't change what runs. |
| 197 | Check out the pull request's code yourself only if you trust it. |
| 198 | - Both report their checks on the pull request's head commit. `opened`, |
| 199 | `synchronize`, `reopened` and `closed` (with `pull_request.merged`) fire. |
| 200 | - Merge queue: turn on **Require a merge queue** in a branch protection rule |
| 201 | (or `"mergeQueue": {"method": "squash"}` through the API). "Merge when ready" |
| 202 | then adds a pull request to the queue. Each entry gets a merge group branch, |
| 203 | `gh-readonly-queue/<base>/pr-<n>-<head sha>`, holding the base, every pull |
| 204 | request ahead of it and its own changes, and `merge_group` (and `push`) |
| 205 | workflows run on it. Once its required checks pass and the entries ahead have |
| 206 | merged, the base fast-forwards to exactly the tested commit. A group that |
| 207 | fails, conflicts or exceeds the check timeout leaves the queue with a comment |
| 208 | saying why, and the groups behind it are rebuilt. The queue is at |
| 209 | `/<owner>/<repo>/queue/<branch>`; the API is |
| 210 | `PUT`/`DELETE /repos/{owner}/{repo}/pulls/{n}/merge-queue` and |
| 211 | `GET /repos/{owner}/{repo}/merge-queue/{branch}`, `diffinery pr merge` queues |
| 212 | automatically, and the MCP `merge_pull_request` tool does too. |
| 213 | |
| 214 | Supported workflow features: |
| 215 | |
| 216 | - `needs` and job outputs, `if` with `success()`, `failure()`, `always()` and |
| 217 | `cancelled()`, `strategy.matrix` with include and exclude, `fail-fast` and |
| 218 | `max-parallel`, `concurrency` with `cancel-in-progress`, and `timeout-minutes`. |
| 219 | - `continue-on-error`, `defaults.run`, `env` at every level, `$GITHUB_OUTPUT`, |
| 220 | `$GITHUB_ENV`, `$GITHUB_PATH`, `$GITHUB_STEP_SUMMARY`, and masking of |
| 221 | secrets and `::add-mask::` values. |
| 222 | - `secrets.*`, `vars.*`, `GITHUB_TOKEN` (a per-job token), and |
| 223 | `permissions: id-token: write` for OIDC (issuer `https://diffinery.com`). |
| 224 | - `runs-on: ubuntu-latest` (and any `ubuntu-*` or `linux` label) runs in a |
| 225 | Cloudflare Container. `runs-on: [self-hosted, …]` jobs wait for a runner you |
| 226 | start with `diffinery runner -R owner/repo --labels gpu,macos`. Use that for |
| 227 | macOS, GPUs or private networks. |
| 228 | - Reusable workflows: `jobs.<id>.uses: ./.github/workflows/x.yml` from the same |
| 229 | commit, or `owner/repo/.github/workflows/x.yml@ref` from another repository |
| 230 | on this instance (public or internal ones, or private ones with the same |
| 231 | owner). `with`, `secrets` (a mapping or `inherit`), `on.workflow_call` |
| 232 | outputs and nesting up to four levels deep all work. A matrix on the calling |
| 233 | job isn't supported yet. |
| 234 | - Docker, on hosted runners and on self-hosted runners with Docker installed: |
| 235 | a job `container:` (steps and JavaScript actions run inside it), `services:` |
| 236 | (reachable by name from the job container, or on `localhost` through their |
| 237 | mapped ports, with health checks awaited), Docker container actions |
| 238 | (`docker://image` steps and actions with `runs.using: docker`, built from |
| 239 | their Dockerfile), and plain `docker build` / `docker run` / `docker compose` |
| 240 | in steps. Hosted runners start the Docker daemon the first time a job uses |
| 241 | it and pull Docker Hub images through `mirror.gcr.io`. |
| 242 | - A container registry at `diffinery.com` in place of ghcr.io: sign in with |
| 243 | the job's `GITHUB_TOKEN` (`permissions: packages: write`) or a personal |
| 244 | token, then push and pull `diffinery.com/<owner>/<name>`. Images are |
| 245 | private by default and can be linked to a repository. See |
| 246 | [docs/container-registry.md](docs/container-registry.md). |
| 247 | |
| 248 | **Not supported yet:** `actions/github-script`, |
| 249 | Windows and macOS hosted runners, and |
| 250 | `deepen-not` / partial clone filters. A job that uses an unsupported feature fails |
| 251 | with a clear message instead of misbehaving. |
| 252 | |
| 253 | ## Developing |
| 254 | |
| 255 | ```sh |
| 256 | pnpm install |
| 257 | pnpm -r build && pnpm -r typecheck && pnpm -r test |
| 258 | pnpm --filter @diffinery/app db:migrate:local |
| 259 | pnpm dev # http://localhost:8787, signed in as dev@diffinery.local |
| 260 | pnpm --filter @diffinery/app e2e # wrangler dev + real git + the runner agent + Chromium |
| 261 | ``` |
| 262 | |
| 263 | The end-to-end test does the following: |
| 264 | |
| 265 | 1. Pushes a repository with real `git`. |
| 266 | 2. Runs its CI through the actual runner agent. |
| 267 | 3. Checks secret scanning and branch protection. |
| 268 | 4. Has an agent open a pull request that needs a human approval and a green |
| 269 | check, then merges it. |
| 270 | 5. Runs pull request workflows on test merge commits, `pull_request_target` |
| 271 | from the base branch, and a merge queue with `merge_group` runs, including a |
| 272 | group that fails and leaves the queue. |
| 273 | 6. Clones shallow, then deepens the clone. |
| 274 | 7. Drives the MCP server. |
| 275 | 8. In Chromium with a virtual authenticator, joins from an invite with a new |
| 276 | passkey, adds and removes passkeys, signs out and in, and uses an admin |
| 277 | sign-in link. |
| 278 | |
| 279 | Team preferences for product and design decisions are in |
| 280 | [docs/PREFERENCES.md](docs/PREFERENCES.md). |