Diffinery

codician/diffinerypublic

Open-source, agent-first GitHub and GitHub Actions alternative that runs on your own Cloudflare account: git hosting, pull requests, code review and drop-in GitHub Actions YAML on Workers, Durable Objects, R2 and Containers.

406 lines · 23.0 KBRaw History
1# Diffinery
2
3A git forge for teams of people and AI agents, running entirely on Cloudflare.
4Git hosting, pull requests and reviews, issues, GitHub-Actions-compatible CI,
5an MCP server for agents, and sandboxes, with no servers to run.
6
7- **Drop-in for GitHub workflows.** `.github/workflows/*.yml` runs as-is: same
8 triggers, expressions, matrices, `needs`, outputs, caches, artifacts, secrets,
9 OIDC, `actions/checkout`, `setup-node/python/go`, and remote JavaScript and
10 composite actions.
11- **Built for agents.** Agents are first-class accounts owned by a person or
12 team, with their own tokens and limits. They can't push or merge to protected
13 branches, and their pull requests always need a human approval. Every commit
14 they make is stamped with who they act for. They work through MCP
15 (`/mcp`) or the REST API, and can start sandboxes to build and test.
16- **Secure by default.** People sign in with passkeys, with no passwords to
17 leak or phish. Invites, roles and branch protection decide what they can do. Pushes are scanned for
18 leaked credentials and blocked if they contain any; nothing from a blocked
19 push is kept. A repository admin (a person, never an agent) can push anyway
20 with `git push -o allow-secrets=<reason>`, which is audited. A file can opt
21 out with a `diffinery:allow-secret` comment, for test fixtures. Secrets use
22 envelope encryption, and every sensitive action goes to an append-only audit log.
23
24## How it fits on Cloudflare
25
26| Piece | Cloudflare product |
27|---|---|
28| HTTP, git smart HTTP, REST API, MCP, web UI | Workers (Hono) |
29| One repository: refs, objects, PRs, issues, reviews, checks, runs | Durable Object with SQLite (`RepoDO`), one per repository |
30| Large objects, Git LFS, container images, logs, caches, artifacts, event archive | R2 |
31| Accounts, orgs, tokens, invites, notifications, audit log | D1, read from nearby replicas through the Sessions API (each browser keeps a bookmark so it reads its own writes) |
32| Webhooks, notifications, audit fan-out | Queues (with a dead-letter queue) |
33| Workflow runs (durable, survive restarts, 6 h job timeout) | Workflows |
34| Job runners and agent sandboxes | Containers (Ubuntu 24.04 image in `containers/runner`) |
35| Scheduled workflows (`on: schedule`) | Cron Triggers |
36| Live run logs and PR updates | Durable Object WebSockets (hibernation) |
37| Sign-in | Passkeys (WebAuthn) and sessions in D1. Cloudflare Access guards only `/auth/access`, the email recovery path |
38
39```
40apps/diffinery the Worker (src/index.ts), D1 migrations, wrangler config, e2e test
41apps/cli the `diffinery` command-line client
42containers/runner runner agent + sandbox server, and the container image
43packages/git-core git objects, packfiles, deltas, diff/merge (pure TypeScript)
44packages/workflow-yaml GitHub Actions workflow parser, expressions, matrix, cron
45packages/shared types shared by the Worker and the runner
46scripts bootstrap.sh, ensure-secrets.sh, render-config.mjs
47```
48
49## Deploying
50
51Diffinery deploys itself: this repository lives at
52`diffinery.com/codician-team/diffinery`, and every push to `main` there runs
53`.github/workflows/deploy.yml` on Diffinery's hosted runners, followed by the
54live test (`.github/workflows/live-test.yml`). GitHub stays the upstream for
55now: `.github/workflows/sync-to-diffinery.yml` copies every branch and tag, and
56the secrets and variables below, to Diffinery on each push, then waits for the
57Actions runs there and fails if any of them fail. Running Deploy by hand on
58GitHub still deploys, as a fallback.
59
60The workflow needs two repository secrets:
61
62- `CLOUDFLARE_API_TOKEN`: account permissions Workers Scripts, D1, R2,
63 Queues, Workflows, Containers and Access: Apps and Policies (all Edit), and
64 zone permission Workers Routes Edit on `diffinery.com`.
65- `CLOUDFLARE_ACCOUNT_ID`
66
67On each deploy, `scripts/bootstrap.sh` creates anything that is missing: the D1
68database, R2 bucket, queues and one Access application that covers only
69`/auth/access` (email-based account recovery). It then applies D1 migrations. `wrangler deploy`
70builds and pushes the runner image. On the first deploy, `ensure-secrets.sh`
71generates `MASTER_KEY` and the OIDC signing key inside the job, so neither
72value is ever printed.
73
74If the token can't read your Zero Trust organization, set a repository
75**variable** `ACCESS_TEAM_DOMAIN` (for example `codician.cloudflareaccess.com`).
76
77To run your own Diffinery on another domain, set `DIFFINERY_DOMAIN` (for
78example `git.example.com`) when running `scripts/bootstrap.sh`. The Worker's
79route, `PUBLIC_URL` and the email sender follow it. Set `DIFFINERY_ZONE` if
80the Cloudflare zone isn't the domain's last two labels (for example
81`example.co.uk`), and `MAIL_DOMAIN` to send email from another domain. The
82token then needs Workers Routes Edit on that zone instead of `diffinery.com`.
83
84Email recovery uses Access's email one-time PIN, which works without any setup.
85You can also add GitHub as a login method under Zero Trust → Settings → Authentication.
86
87## Signing in and inviting people
88
89Everyone signs in with a passkey. There are no passwords.
90
911. **First sign-in for admins.** The emails in `BOOTSTRAP_ADMINS` (in
92 `apps/diffinery/wrangler.jsonc`) become site admins. Open
93 `https://diffinery.com/auth/access`, confirm your email with the one-time
94 code, and add a passkey on the page that follows.
952. **Inviting someone.** An admin or org owner creates an invite from
96 *Settings → Site administration*, an organization's *People* tab, or
97 `diffinery invite friend@example.com --org codician`. Diffinery emails them
98 the link; they open it, pick a username and create a passkey, and they're in.
993. **Lost passkey.** A site admin clicks *Sign-in link* next to the person in
100 *Settings → Site administration*; Diffinery emails them the one-time link,
101 which works for 24 hours. People can also recover through `/auth/access` by email.
1024. **More devices.** Add passkeys under *Settings → Passkeys and sessions*, or
103 keep one in a password manager that syncs. The same page signs out other
104 browsers.
1055. Repository and org roles (read, triage, write, maintain, admin) decide what
106 each person can do.
107
108Nobody needs a Cloudflare account. Git and the CLI use a personal token from
109*Settings → Tokens*.
110
111```sh
112git clone https://diffinery.com/codician/website.git # username: anything, password: your dfy_ token
113diffinery login && diffinery auth setup-git # or let the CLI handle credentials
114```
115
116## Email
117
118Diffinery sends email through [Cloudflare Email Sending](https://developers.cloudflare.com/email-service/)
119from `notifications@diffinery.com` (`MAIL_FROM` in `wrangler.jsonc`). Every
120email has an HTML and a plain-text version, and they work like GitHub's:
121
122| Group | Emails | Can be turned off |
123| --- | --- | --- |
124| Security | New sign-in from an unknown browser or device, passkey added or removed, access token created or about to expire, agent added, push protection bypassed or a push blocked | No |
125| Account | Invitations, sign-in links, welcome, account disabled or enabled, test email | No |
126| Participating | Issues and pull requests you opened, commented on, reviewed, were assigned, asked to review, @mentioned in or subscribed to: new comments, reviews, pushes, merges, closes and reopens | Yes |
127| Watching | Everything in repositories you watch with *All activity* | Yes |
128| Actions | Failed workflow runs you started (all results if you ask) | Yes |
129| Repositories | Repositories created or deleted in your organizations, repository access granted | Yes |
130| Organizations | Organization role changes, accepted invitations, access requests (admins) | Yes |
131
132- **Settings.** *Settings → Notifications* turns each group on or off, on
133 Diffinery and by email, lists the repositories you watch and sends a test
134 email. Each repository has a *Watch* menu (participating, all activity,
135 ignore) and each issue and pull request a *Subscribe / Unsubscribe* button.
136 Your own comments and changes don't email you unless you ask.
137- **Unsubscribing.** Emails carry `List-Unsubscribe` and one-click
138 unsubscribe headers (RFC 8058), so mail apps show an Unsubscribe button, and
139 the footer links mute the thread, stop watching the repository or turn off
140 that group. Links are signed, so they work without signing in.
141- **Threading.** Emails about one issue or pull request share
142 `References`/`In-Reply-To` headers and a `[owner/repo]` subject, and carry
143 `X-Diffinery-Reason` (like `X-GitHub-Reason`) for filters.
144- **Delivery.** Emails go through the `diffinery-mail` queue (retries with
145 backoff, then the `diffinery-mail-dlq` dead-letter queue) and are logged for
146 90 days. Site admins see delivery status, the log and a preview of every
147 template under *Settings → Email*, and can send every template to
148 themselves.
149- **API.** `GET/PUT/DELETE /api/v3/repos/{owner}/{repo}/subscription`,
150 `GET/PUT /api/v3/diffinery/notification-settings`,
151 `GET /api/v3/diffinery/emails`, `POST /api/v3/diffinery/emails/test`; for
152 site admins `GET /api/v3/diffinery/admin/email` and
153 `POST /api/v3/diffinery/admin/emails/samples`.
154
155**Setting it up.** Onboard the sending domain once in the Cloudflare dashboard
156(*Compute → Email Service → Email Sending → Onboard Domain*); Cloudflare adds
157the bounce MX, SPF, DKIM and DMARC records. Each deploy runs
158`scripts/email-sending.sh`, which checks those records (with the *Email
159Sending: Read* permission on the API token, or the public DNS records without
160it) and warns if mail can't be delivered. In `wrangler dev`, Miniflare accepts
161emails without sending them; `GET /api/v3/diffinery/emails` shows the full
162message there.
163
164## Agents
165
166Create an agent under *Settings → Agents* or with `diffinery agent create reviewer-bot --org codician`.
167Then point any MCP client at it:
168
169```json
170{ "mcpServers": { "diffinery": { "type": "http", "url": "https://diffinery.com/mcp", "headers": { "Authorization": "Bearer dfy_…" } } } }
171```
172
173Tools include `get_file`, `list_tree`, `commit_files` (no git client needed),
174`create_branch`, `create_pull_request` (with an `intent` for reviewers),
175`review_pull_request`, `merge_pull_request` (policy-guarded), issues,
176`list_workflow_runs` and `get_job_logs`, `rerun_workflow`, releases
177(`create_release`, `upload_release_asset`, `generate_release_notes`), and
178`sandbox_create`/`sandbox_exec` for running builds and tests.
179
180Agent guardrails apply everywhere: git, REST, MCP and the web UI.
181
182- An agent's role is capped at write, and never exceeds its owner's role.
183- Agents can't push to the default branch or protected branches, and can't
184 merge there. Turn this on per branch with `agentsMayPush` or `agentsMayMerge`.
185- A pull request opened by an agent needs at least one human approval, and
186 approvals from agents don't count toward it.
187- Agents can't change branch protection or read or set secrets.
188- Agents can't change or delete environments or their branch policies, and
189 can't dismiss reviews.
190- Commit statuses and check runs that agents report never satisfy a required
191 check.
192
193## Workflows
194
195Put workflow files in `.github/workflows/` (or `.diffinery/workflows/`).
196Supported triggers are `push`, `pull_request`, `pull_request_target`,
197`merge_group`, `workflow_dispatch` (with inputs), `schedule` and `workflow_run`
198(up to three levels deep, like GitHub), with branch, tag and path filters.
199
200Pull requests behave as on GitHub:
201
202- `pull_request` runs check out the test merge commit, `refs/pull/<n>/merge`
203 (the head merged into the current base), with `GITHUB_REF` set to that ref.
204 While a pull request conflicts with its base, its `pull_request` workflows
205 don't run. `refs/pull/<n>/head` and `refs/pull/<n>/merge` can be fetched but
206 not pushed.
207- `pull_request_target` runs the base branch's own workflow on the base commit,
208 with the repository's secrets, so a pull request can't change what runs.
209 Check out the pull request's code yourself only if you trust it.
210- Both report their checks on the pull request's head commit. `opened`,
211 `synchronize`, `reopened` and `closed` (with `pull_request.merged`) fire.
212- Merge queue: turn on **Require a merge queue** in a branch protection rule
213 (or `"mergeQueue": {"method": "squash"}` through the API). "Merge when ready"
214 then adds a pull request to the queue. Each entry gets a merge group branch,
215 `gh-readonly-queue/<base>/pr-<n>-<head sha>`, holding the base, every pull
216 request ahead of it and its own changes, and `merge_group` (and `push`)
217 workflows run on it. Once its required checks pass and the entries ahead have
218 merged, the base fast-forwards to exactly the tested commit. A group that
219 fails, conflicts or exceeds the check timeout leaves the queue with a comment
220 saying why, and the groups behind it are rebuilt. The queue is at
221 `/<owner>/<repo>/queue/<branch>`; the API is
222 `PUT`/`DELETE /repos/{owner}/{repo}/pulls/{n}/merge-queue` and
223 `GET /repos/{owner}/{repo}/merge-queue/{branch}`, `diffinery pr merge` queues
224 automatically, and the MCP `merge_pull_request` tool does too.
225
226Supported workflow features:
227
228- `needs` and job outputs, `if` with `success()`, `failure()`, `always()` and
229 `cancelled()`, `strategy.matrix` with include and exclude, `fail-fast` and
230 `max-parallel`, `concurrency` with `cancel-in-progress`, and `timeout-minutes`.
231- `continue-on-error`, `defaults.run`, `env` at every level, `$GITHUB_OUTPUT`,
232 `$GITHUB_ENV`, `$GITHUB_PATH`, `$GITHUB_STEP_SUMMARY`, and masking of
233 secrets and `::add-mask::` values.
234- `secrets.*`, `vars.*`, `GITHUB_TOKEN` (a per-job token), and
235 `permissions: id-token: write` for OIDC (issuer `https://diffinery.com`).
236- `runs-on: ubuntu-latest` (and any `ubuntu-*` or `linux` label) runs in a
237 Cloudflare Container. `runs-on: [self-hosted, …]` jobs wait for a runner you
238 start with `diffinery runner -R owner/repo --labels gpu,macos`. Use that for
239 macOS, GPUs or private networks.
240- Reusable workflows: `jobs.<id>.uses: ./.github/workflows/x.yml` from the same
241 commit, or `owner/repo/.github/workflows/x.yml@ref` from another repository
242 on this instance (public ones, or private ones with the same
243 owner). `with`, `secrets` (a mapping or `inherit`), `on.workflow_call`
244 outputs and nesting up to four levels deep all work. A matrix on the calling
245 job isn't supported yet.
246- Docker, on hosted runners and on self-hosted runners with Docker installed:
247 a job `container:` (steps and JavaScript actions run inside it), `services:`
248 (reachable by name from the job container, or on `localhost` through their
249 mapped ports, with health checks awaited), Docker container actions
250 (`docker://image` steps and actions with `runs.using: docker`, built from
251 their Dockerfile), and plain `docker build` / `docker run` / `docker compose`
252 in steps. Hosted runners start the Docker daemon the first time a job uses
253 it and pull Docker Hub images through `mirror.gcr.io`.
254- A container registry at `diffinery.com` in place of ghcr.io: sign in with
255 the job's `GITHUB_TOKEN` (`permissions: packages: write`) or a personal
256 token, then push and pull `diffinery.com/<owner>/<name>`. Images are
257 private by default and can be linked to a repository. See
258 [docs/container-registry.md](docs/container-registry.md).
259
260**Not supported yet:** `actions/github-script`,
261Windows and macOS hosted runners, and
262`deepen-not` / partial clone filters. A job that uses an unsupported feature fails
263with a clear message instead of misbehaving.
264
265## REST API
266
267The API lives at `/api/v3`, with the same paths, JSON shapes and errors as
268GitHub's REST API. Point a client at it and it works:
269
270```js
271const octokit = new Octokit({ auth: token, baseUrl: "https://diffinery.example.com/api/v3" });
272```
273
274Responses carry `node_id`, `url`, `html_url`, full user objects and label
275objects. Lists take `per_page` and `page` and send a `Link` header, so
276`octokit.paginate` works. Errors have `message` and `documentation_url`.
277Responses also keep a few Diffinery fields (such as `merge_blockers` on pull
278requests) for the CLI, MCP and web UI.
279
280What works:
281
282- **Repositories:** get, create, update, delete, topics, languages,
283 collaborators, branches, tags, readme, contents (GET, and PUT and DELETE
284 with `sha`; a stale `sha` returns 409), compare, commits, `merges`,
285 `dispatches`, and `zipball`/`tarball` archives. `private: false` makes a
286 repository public.
287- **Archives:** `zipball` and `tarball` are built on demand and cached in R2.
288 The cache can be evicted at any time; the archive is then built again. An
289 archive holds at most 24 MB of files and 100,000 tree entries. A zip holds
290 at most 65,535 files; use the tarball for more. Each repository can build
291 20 new archives per hour. Clone bigger repositories with git.
292- **Git data:** blobs, trees (with `base_tree` and `recursive`), commits,
293 annotated tags, `git/ref`, `git/refs` and `git/matching-refs`.
294- **Releases:** list, get by id or tag, latest, create (the tag is created
295 if missing), update, delete, `generate-notes` (honours
296 `.github/release.yml`), `make_latest`, and drafts (visible to writers only).
297 Upload assets to the release's `upload_url`. Download them from
298 `browser_download_url`, or from the asset URL with
299 `Accept: application/octet-stream`. Bytes live in R2. An asset can be up to
300 100 MB, the largest request body Workers accept.
301- **Commit statuses:** create, list and the combined status. A status counts
302 as a check, so a branch protection's required checks can name a status
303 context.
304- **Required checks:** when Diffinery Actions reports a check, it alone
305 decides that check. To have a required check decided by someone else, bind
306 it in branch protection, for example
307 `"requiredCheckSources": { "ci/external": "ci-bot" }`. A required check
308 fails if any counted report of it fails. Only the reporter that created a
309 check run can update it, and Actions check runs can't be changed over REST.
310- **Checks:** check runs (create, update, get, annotations) and check suites.
311- **Issues and pull requests:** labels, assignees, lock and unlock,
312 comments (by issue and repository-wide), reactions (issues, comments, review
313 comments and releases), events and the timeline, reviews, review comments
314 and replies, review dismissal, requested reviewers, `update-branch`, merge,
315 `commits/{sha}/pulls`, and list filters (`head`, `base`, `sort`,
316 `direction`, `labels`, `assignee`, `since`). Pull requests report
317 `mergeable`, `mergeable_state` and `rebaseable`. Comment, review, check
318 run, user and repository ids are numbers, as on GitHub. Locked
319 conversations refuse comments, review comments and reactions from people
320 without triage access. Dismissing a review takes maintain access, and the
321 pull request's author can't do it.
322- **Drafts and merging:** `PATCH /pulls/{n}` with `"draft": false` marks a
323 draft ready for review (`"draft": true` turns it back; GitHub only offers
324 this through GraphQL). The web page has a "Ready for review" button and the
325 CLI has `diffinery pr ready`. A merge commit is refused on a branch that
326 requires linear history. Admins can merge past unmet branch rules only by
327 sending `"bypass_rules": true` (the CLI's `pr merge --admin`); the bypass is
328 written to the audit log. When several rules match a branch, the rule that
329 names it exactly wins, then the oldest pattern.
330- **Who can write:** anonymous callers can read public repositories but can't
331 create or change anything. A `read`-scoped token only reads; creating
332 organizations, invitations, agents and tokens takes an `admin`-scoped token
333 or a browser session. Token scopes must be `read`, `write` or `admin`.
334 Agents can't add, change or delete secrets or variables, whatever their
335 role.
336- **Archived repositories** are read-only through the API, for everyone.
337- **Tree entry names:** every write refuses `.`, `..`, `.git` (in any case,
338 and spellings like `.git.` or `git~1`), and names with `\` or NUL.
339- **Deployments and environments:** deployments, deployment statuses,
340 environments, deployment branch policies, and environment secrets and
341 variables. A deployment without `required_contexts` needs every context on
342 the commit to have passed, as on GitHub; send `[]` to skip that. An
343 environment with no branch policy gets "protected branches only" when its
344 first secret is added. Environment secrets never go to `refs/pull/*` runs.
345 Runs started by agents only get them when the environment allows protected
346 branches only. Tags pushed by agents never match an environment's tag
347 patterns.
348- **Actions:** runs, run attempts, jobs, logs (a zip per run), artifacts,
349 workflows, secrets and variables.
350- **Search:** `/search/issues` (`repo:`, `is:`, `label:`, `author:`,
351 `assignee:`, `no:`, `head:`, `base:`, `in:`, date ranges and `sort:`) and
352 `/search/repositories` (`user:`, `org:`, `topic:`, `is:public`,
353 `archived:`). Search only covers repositories you can read. A query
354 can be up to 256 characters, and one search looks at up to 50 repositories.
355- `/rate_limit`. Diffinery doesn't rate-limit the API, so it reports a
356 generous budget.
357
358Workflows can react to these too. A published release triggers `on: release`,
359a `POST /dispatches` triggers `on: repository_dispatch`, and webhooks get
360`release`, `status`, `deployment`, `deployment_status` and
361`workflow_dispatch` events, and `edited`, `ready_for_review` and
362`converted_to_draft` actions on issues and pull requests. A delivery that
363fails with a network error, 408, 429 or 5xx is retried up to five times over
364about five hours. A job with
365`environment:` creates a deployment, gets that environment's secrets and
366variables, and is refused if the environment's branch policy doesn't allow
367the ref. As on GitHub, events caused by `GITHUB_TOKEN` don't start new
368workflow runs, except `repository_dispatch`.
369
370**Not supported yet:** GraphQL (`/api/graphql`), setting secrets with a
371sealed-box `encrypted_value` (send `value` instead), environment protection
372rules (required reviewers, wait timers), forks, team reviewers, and code and
373commit search. Search qualifiers for stars, forks and languages are accepted
374but ignored.
375
376## Developing
377
378```sh
379pnpm install
380pnpm -r build && pnpm -r typecheck && pnpm -r test
381pnpm --filter @diffinery/app db:migrate:local
382pnpm dev # http://localhost:8787, signed in as dev@diffinery.local
383pnpm --filter @diffinery/app e2e # wrangler dev + real git + the runner agent + Chromium
384```
385
386The end-to-end test does the following:
387
3881. Pushes a repository with real `git`.
3892. Runs its CI through the actual runner agent.
3903. Checks secret scanning and branch protection.
3914. Has an agent open a pull request that needs a human approval and a green
392 check, then merges it.
3935. Runs pull request workflows on test merge commits, `pull_request_target`
394 from the base branch, and a merge queue with `merge_group` runs, including a
395 group that fails and leaves the queue.
3966. Clones shallow, then deepens the clone.
3977. Drives the MCP server.
3988. In Chromium with a virtual authenticator, joins from an invite with a new
399 passkey, adds and removes passkeys, signs out and in, and uses an admin
400 sign-in link.
4019. Drives the REST API with the real `@octokit/rest` client: contents, git
402 data, statuses as required checks, releases with asset uploads and
403 downloads, issues, reactions, pagination, search and deployments.
404
405Team preferences for product and design decisions are in
406[docs/PREFERENCES.md](docs/PREFERENCES.md).