Diffinery

codician/diffinerypublic

Open-source, agent-first GitHub and GitHub Actions alternative that runs on your own Cloudflare account: git hosting, pull requests, code review and drop-in GitHub Actions YAML on Workers, Durable Objects, R2 and Containers.

381 lines · 21.4 KBRaw History
1# Diffinery
2
3A git forge for teams of people and AI agents, running entirely on Cloudflare.
4Git hosting, pull requests and reviews, issues, GitHub-Actions-compatible CI,
5an MCP server for agents, and sandboxes, with no servers to run.
6
7- **Drop-in for GitHub workflows.** `.github/workflows/*.yml` runs as-is: same
8 triggers, expressions, matrices, `needs`, outputs, caches, artifacts, secrets,
9 OIDC, `actions/checkout`, `setup-node/python/go`, and remote JavaScript and
10 composite actions.
11- **Built for agents.** Agents are first-class accounts owned by a person or
12 team, with their own tokens and limits. They can't push or merge to protected
13 branches, and their pull requests always need a human approval. Every commit
14 they make is stamped with who they act for. They work through MCP
15 (`/mcp`) or the REST API, and can start sandboxes to build and test.
16- **Secure by default.** People sign in with passkeys, with no passwords to
17 leak or phish. Invites, roles and branch protection decide what they can do. Pushes are scanned for
18 leaked credentials and blocked if they contain any; nothing from a blocked
19 push is kept. A repository admin (a person, never an agent) can push anyway
20 with `git push -o allow-secrets=<reason>`, which is audited. A file can opt
21 out with a `diffinery:allow-secret` comment, for test fixtures. Secrets use
22 envelope encryption, and every sensitive action goes to an append-only audit log.
23
24## How it fits on Cloudflare
25
26| Piece | Cloudflare product |
27|---|---|
28| HTTP, git smart HTTP, REST API, MCP, web UI | Workers (Hono) |
29| One repository: refs, objects, PRs, issues, reviews, checks, runs | Durable Object with SQLite (`RepoDO`), one per repository |
30| Large objects, Git LFS, container images, logs, caches, artifacts, event archive | R2 |
31| Accounts, orgs, tokens, invites, notifications, audit log | D1, read from nearby replicas through the Sessions API (each browser keeps a bookmark so it reads its own writes) |
32| Webhooks, notifications, audit fan-out | Queues (with a dead-letter queue) |
33| Workflow runs (durable, survive restarts, 6 h job timeout) | Workflows |
34| Job runners and agent sandboxes | Containers (Ubuntu 24.04 image in `containers/runner`) |
35| Scheduled workflows (`on: schedule`) | Cron Triggers |
36| Live run logs and PR updates | Durable Object WebSockets (hibernation) |
37| Sign-in | Passkeys (WebAuthn) and sessions in D1. Cloudflare Access guards only `/auth/access`, the email recovery path |
38
39```
40apps/diffinery the Worker (src/index.ts), D1 migrations, wrangler config, e2e test
41apps/cli the `diffinery` command-line client
42containers/runner runner agent + sandbox server, and the container image
43packages/git-core git objects, packfiles, deltas, diff/merge (pure TypeScript)
44packages/workflow-yaml GitHub Actions workflow parser, expressions, matrix, cron
45packages/shared types shared by the Worker and the runner
46scripts bootstrap.sh, ensure-secrets.sh, render-config.mjs
47```
48
49## Deploying
50
51Diffinery deploys itself: this repository lives at
52`diffinery.com/codician-team/diffinery`, and every push to `main` there runs
53`.github/workflows/deploy.yml` on Diffinery's hosted runners, followed by the
54live test (`.github/workflows/live-test.yml`). GitHub stays the upstream for
55now: `.github/workflows/sync-to-diffinery.yml` copies every branch and tag, and
56the secrets and variables below, to Diffinery on each push, then waits for the
57Actions runs there and fails if any of them fail. Running Deploy by hand on
58GitHub still deploys, as a fallback.
59
60The workflow needs two repository secrets:
61
62- `CLOUDFLARE_API_TOKEN`: account permissions Workers Scripts, D1, R2,
63 Queues, Workflows, Containers and Access: Apps and Policies (all Edit), and
64 zone permission Workers Routes Edit on `diffinery.com`.
65- `CLOUDFLARE_ACCOUNT_ID`
66
67On each deploy, `scripts/bootstrap.sh` creates anything that is missing: the D1
68database, R2 bucket, queues and one Access application that covers only
69`/auth/access` (email-based account recovery). It then applies D1 migrations. `wrangler deploy`
70builds and pushes the runner image. On the first deploy, `ensure-secrets.sh`
71generates `MASTER_KEY` and the OIDC signing key inside the job, so neither
72value is ever printed.
73
74If the token can't read your Zero Trust organization, set a repository
75**variable** `ACCESS_TEAM_DOMAIN` (for example `codician.cloudflareaccess.com`).
76
77Email recovery uses Access's email one-time PIN, which works without any setup.
78You can also add GitHub as a login method under Zero Trust → Settings → Authentication.
79
80## Signing in and inviting people
81
82Everyone signs in with a passkey. There are no passwords.
83
841. **First sign-in for admins.** The emails in `BOOTSTRAP_ADMINS` (in
85 `apps/diffinery/wrangler.jsonc`) become site admins. Open
86 `https://diffinery.com/auth/access`, confirm your email with the one-time
87 code, and add a passkey on the page that follows.
882. **Inviting someone.** An admin or org owner creates an invite from
89 *Settings → Site administration*, an organization's *People* tab, or
90 `diffinery invite friend@example.com --org codician`. Diffinery emails them
91 the link; they open it, pick a username and create a passkey, and they're in.
923. **Lost passkey.** A site admin clicks *Sign-in link* next to the person in
93 *Settings → Site administration*; Diffinery emails them the one-time link,
94 which works for 24 hours. People can also recover through `/auth/access` by email.
954. **More devices.** Add passkeys under *Settings → Passkeys and sessions*, or
96 keep one in a password manager that syncs. The same page signs out other
97 browsers.
985. Repository and org roles (read, triage, write, maintain, admin) decide what
99 each person can do.
100
101Nobody needs a Cloudflare account. Git and the CLI use a personal token from
102*Settings → Tokens*.
103
104```sh
105git clone https://diffinery.com/codician/website.git # username: anything, password: your dfy_ token
106diffinery login && diffinery auth setup-git # or let the CLI handle credentials
107```
108
109## Email
110
111Diffinery sends email through [Cloudflare Email Sending](https://developers.cloudflare.com/email-service/)
112from `notifications@diffinery.com` (`MAIL_FROM` in `wrangler.jsonc`). Every
113email has an HTML and a plain-text version, and they work like GitHub's:
114
115| Group | Emails | Can be turned off |
116| --- | --- | --- |
117| Security | New sign-in from an unknown browser or device, passkey added or removed, access token created or about to expire, agent added, push protection bypassed or a push blocked | No |
118| Account | Invitations, sign-in links, welcome, account disabled or enabled, test email | No |
119| Participating | Issues and pull requests you opened, commented on, reviewed, were assigned, asked to review, @mentioned in or subscribed to: new comments, reviews, pushes, merges, closes and reopens | Yes |
120| Watching | Everything in repositories you watch with *All activity* | Yes |
121| Actions | Failed workflow runs you started (all results if you ask) | Yes |
122| Repositories | Repositories created or deleted in your organizations, repository access granted | Yes |
123| Organizations | Organization role changes, accepted invitations, access requests (admins) | Yes |
124
125- **Settings.** *Settings → Notifications* turns each group on or off, on
126 Diffinery and by email, lists the repositories you watch and sends a test
127 email. Each repository has a *Watch* menu (participating, all activity,
128 ignore) and each issue and pull request a *Subscribe / Unsubscribe* button.
129 Your own comments and changes don't email you unless you ask.
130- **Unsubscribing.** Emails carry `List-Unsubscribe` and one-click
131 unsubscribe headers (RFC 8058), so mail apps show an Unsubscribe button, and
132 the footer links mute the thread, stop watching the repository or turn off
133 that group. Links are signed, so they work without signing in.
134- **Threading.** Emails about one issue or pull request share
135 `References`/`In-Reply-To` headers and a `[owner/repo]` subject, and carry
136 `X-Diffinery-Reason` (like `X-GitHub-Reason`) for filters.
137- **Delivery.** Emails go through the `diffinery-mail` queue (retries with
138 backoff, then the `diffinery-mail-dlq` dead-letter queue) and are logged for
139 90 days. Site admins see delivery status, the log and a preview of every
140 template under *Settings → Email*, and can send every template to
141 themselves.
142- **API.** `GET/PUT/DELETE /api/v3/repos/{owner}/{repo}/subscription`,
143 `GET/PUT /api/v3/diffinery/notification-settings`,
144 `GET /api/v3/diffinery/emails`, `POST /api/v3/diffinery/emails/test`; for
145 site admins `GET /api/v3/diffinery/admin/email` and
146 `POST /api/v3/diffinery/admin/emails/samples`.
147
148**Setting it up.** Onboard the sending domain once in the Cloudflare dashboard
149(*Compute → Email Service → Email Sending → Onboard Domain*); Cloudflare adds
150the bounce MX, SPF, DKIM and DMARC records. Each deploy runs
151`scripts/email-sending.sh`, which checks those records (with the *Email
152Sending: Read* permission on the API token, or the public DNS records without
153it) and warns if mail can't be delivered. In `wrangler dev`, Miniflare accepts
154emails without sending them; `GET /api/v3/diffinery/emails` shows the full
155message there.
156
157## Agents
158
159Create an agent under *Settings → Agents* or with `diffinery agent create reviewer-bot --org codician`.
160Then point any MCP client at it:
161
162```json
163{ "mcpServers": { "diffinery": { "type": "http", "url": "https://diffinery.com/mcp", "headers": { "Authorization": "Bearer dfy_…" } } } }
164```
165
166Tools include `get_file`, `list_tree`, `commit_files` (no git client needed),
167`create_branch`, `create_pull_request` (with an `intent` for reviewers),
168`review_pull_request`, `merge_pull_request` (policy-guarded), issues,
169`list_workflow_runs` and `get_job_logs`, `rerun_workflow`, releases
170(`create_release`, `upload_release_asset`, `generate_release_notes`), and
171`sandbox_create`/`sandbox_exec` for running builds and tests.
172
173Agent guardrails apply everywhere: git, REST, MCP and the web UI.
174
175- An agent's role is capped at write, and never exceeds its owner's role.
176- Agents can't push to the default branch or protected branches, and can't
177 merge there. Turn this on per branch with `agentsMayPush` or `agentsMayMerge`.
178- A pull request opened by an agent needs at least one human approval, and
179 approvals from agents don't count toward it.
180- Agents can't change branch protection or read or set secrets.
181- Agents can't change or delete environments or their branch policies, and
182 can't dismiss reviews.
183- Commit statuses and check runs that agents report never satisfy a required
184 check.
185
186## Workflows
187
188Put workflow files in `.github/workflows/` (or `.diffinery/workflows/`).
189Supported triggers are `push`, `pull_request`, `pull_request_target`,
190`merge_group`, `workflow_dispatch` (with inputs), `schedule` and `workflow_run`
191(up to three levels deep, like GitHub), with branch, tag and path filters.
192
193Pull requests behave as on GitHub:
194
195- `pull_request` runs check out the test merge commit, `refs/pull/<n>/merge`
196 (the head merged into the current base), with `GITHUB_REF` set to that ref.
197 While a pull request conflicts with its base, its `pull_request` workflows
198 don't run. `refs/pull/<n>/head` and `refs/pull/<n>/merge` can be fetched but
199 not pushed.
200- `pull_request_target` runs the base branch's own workflow on the base commit,
201 with the repository's secrets, so a pull request can't change what runs.
202 Check out the pull request's code yourself only if you trust it.
203- Both report their checks on the pull request's head commit. `opened`,
204 `synchronize`, `reopened` and `closed` (with `pull_request.merged`) fire.
205- Merge queue: turn on **Require a merge queue** in a branch protection rule
206 (or `"mergeQueue": {"method": "squash"}` through the API). "Merge when ready"
207 then adds a pull request to the queue. Each entry gets a merge group branch,
208 `gh-readonly-queue/<base>/pr-<n>-<head sha>`, holding the base, every pull
209 request ahead of it and its own changes, and `merge_group` (and `push`)
210 workflows run on it. Once its required checks pass and the entries ahead have
211 merged, the base fast-forwards to exactly the tested commit. A group that
212 fails, conflicts or exceeds the check timeout leaves the queue with a comment
213 saying why, and the groups behind it are rebuilt. The queue is at
214 `/<owner>/<repo>/queue/<branch>`; the API is
215 `PUT`/`DELETE /repos/{owner}/{repo}/pulls/{n}/merge-queue` and
216 `GET /repos/{owner}/{repo}/merge-queue/{branch}`, `diffinery pr merge` queues
217 automatically, and the MCP `merge_pull_request` tool does too.
218
219Supported workflow features:
220
221- `needs` and job outputs, `if` with `success()`, `failure()`, `always()` and
222 `cancelled()`, `strategy.matrix` with include and exclude, `fail-fast` and
223 `max-parallel`, `concurrency` with `cancel-in-progress`, and `timeout-minutes`.
224- `continue-on-error`, `defaults.run`, `env` at every level, `$GITHUB_OUTPUT`,
225 `$GITHUB_ENV`, `$GITHUB_PATH`, `$GITHUB_STEP_SUMMARY`, and masking of
226 secrets and `::add-mask::` values.
227- `secrets.*`, `vars.*`, `GITHUB_TOKEN` (a per-job token), and
228 `permissions: id-token: write` for OIDC (issuer `https://diffinery.com`).
229- `runs-on: ubuntu-latest` (and any `ubuntu-*` or `linux` label) runs in a
230 Cloudflare Container. `runs-on: [self-hosted, …]` jobs wait for a runner you
231 start with `diffinery runner -R owner/repo --labels gpu,macos`. Use that for
232 macOS, GPUs or private networks.
233- Reusable workflows: `jobs.<id>.uses: ./.github/workflows/x.yml` from the same
234 commit, or `owner/repo/.github/workflows/x.yml@ref` from another repository
235 on this instance (public ones, or private ones with the same
236 owner). `with`, `secrets` (a mapping or `inherit`), `on.workflow_call`
237 outputs and nesting up to four levels deep all work. A matrix on the calling
238 job isn't supported yet.
239- Docker, on hosted runners and on self-hosted runners with Docker installed:
240 a job `container:` (steps and JavaScript actions run inside it), `services:`
241 (reachable by name from the job container, or on `localhost` through their
242 mapped ports, with health checks awaited), Docker container actions
243 (`docker://image` steps and actions with `runs.using: docker`, built from
244 their Dockerfile), and plain `docker build` / `docker run` / `docker compose`
245 in steps. Hosted runners start the Docker daemon the first time a job uses
246 it and pull Docker Hub images through `mirror.gcr.io`.
247- A container registry at `diffinery.com` in place of ghcr.io: sign in with
248 the job's `GITHUB_TOKEN` (`permissions: packages: write`) or a personal
249 token, then push and pull `diffinery.com/<owner>/<name>`. Images are
250 private by default and can be linked to a repository. See
251 [docs/container-registry.md](docs/container-registry.md).
252
253**Not supported yet:** `actions/github-script`,
254Windows and macOS hosted runners, and
255`deepen-not` / partial clone filters. A job that uses an unsupported feature fails
256with a clear message instead of misbehaving.
257
258## REST API
259
260The API lives at `/api/v3`, with the same paths, JSON shapes and errors as
261GitHub's REST API. Point a client at it and it works:
262
263```js
264const octokit = new Octokit({ auth: token, baseUrl: "https://diffinery.example.com/api/v3" });
265```
266
267Responses carry `node_id`, `url`, `html_url`, full user objects and label
268objects. Lists take `per_page` and `page` and send a `Link` header, so
269`octokit.paginate` works. Errors have `message` and `documentation_url`.
270Responses also keep a few Diffinery fields (such as `merge_blockers` on pull
271requests) for the CLI, MCP and web UI.
272
273What works:
274
275- **Repositories:** get, create, update, delete, topics, languages,
276 collaborators, branches, tags, readme, contents (GET, and PUT and DELETE
277 with `sha`; a stale `sha` returns 409), compare, commits, `merges`,
278 `dispatches`, and `zipball`/`tarball` archives. `private: false` makes a
279 repository public.
280- **Archives:** `zipball` and `tarball` are built on demand and cached in R2.
281 The cache can be evicted at any time; the archive is then built again. An
282 archive holds at most 24 MB of files and 100,000 tree entries. A zip holds
283 at most 65,535 files; use the tarball for more. Each repository can build
284 20 new archives per hour. Clone bigger repositories with git.
285- **Git data:** blobs, trees (with `base_tree` and `recursive`), commits,
286 annotated tags, `git/ref`, `git/refs` and `git/matching-refs`.
287- **Releases:** list, get by id or tag, latest, create (the tag is created
288 if missing), update, delete, `generate-notes` (honours
289 `.github/release.yml`), `make_latest`, and drafts (visible to writers only).
290 Upload assets to the release's `upload_url`. Download them from
291 `browser_download_url`, or from the asset URL with
292 `Accept: application/octet-stream`. Bytes live in R2. An asset can be up to
293 100 MB, the largest request body Workers accept.
294- **Commit statuses:** create, list and the combined status. A status counts
295 as a check, so a branch protection's required checks can name a status
296 context.
297- **Required checks:** when Diffinery Actions reports a check, it alone
298 decides that check. To have a required check decided by someone else, bind
299 it in branch protection, for example
300 `"requiredCheckSources": { "ci/external": "ci-bot" }`. A required check
301 fails if any counted report of it fails. Only the reporter that created a
302 check run can update it, and Actions check runs can't be changed over REST.
303- **Checks:** check runs (create, update, get, annotations) and check suites.
304- **Issues and pull requests:** labels, assignees, lock and unlock,
305 comments (by issue and repository-wide), reactions (issues, comments, review
306 comments and releases), events and the timeline, reviews, review comments
307 and replies, review dismissal, requested reviewers, `update-branch`, merge,
308 `commits/{sha}/pulls`, and list filters (`head`, `base`, `sort`,
309 `direction`, `labels`, `assignee`, `since`). Pull requests report
310 `mergeable`, `mergeable_state` and `rebaseable`. Comment, review, check
311 run, user and repository ids are numbers, as on GitHub. Locked
312 conversations refuse comments, review comments and reactions from people
313 without triage access. Dismissing a review takes maintain access, and the
314 pull request's author can't do it.
315- **Archived repositories** are read-only through the API, for everyone.
316- **Tree entry names:** every write refuses `.`, `..`, `.git` (in any case,
317 and spellings like `.git.` or `git~1`), and names with `\` or NUL.
318- **Deployments and environments:** deployments, deployment statuses,
319 environments, deployment branch policies, and environment secrets and
320 variables. A deployment without `required_contexts` needs every context on
321 the commit to have passed, as on GitHub; send `[]` to skip that. An
322 environment with no branch policy gets "protected branches only" when its
323 first secret is added. Environment secrets never go to `refs/pull/*` runs.
324 Runs started by agents only get them when the environment allows protected
325 branches only. Tags pushed by agents never match an environment's tag
326 patterns.
327- **Actions:** runs, run attempts, jobs, logs (a zip per run), artifacts,
328 workflows, secrets and variables.
329- **Search:** `/search/issues` (`repo:`, `is:`, `label:`, `author:`,
330 `assignee:`, `no:`, `head:`, `base:`, `in:`, date ranges and `sort:`) and
331 `/search/repositories` (`user:`, `org:`, `topic:`, `is:public`,
332 `archived:`). Search only covers repositories you can read. A query
333 can be up to 256 characters, and one search looks at up to 50 repositories.
334- `/rate_limit`. Diffinery doesn't rate-limit the API, so it reports a
335 generous budget.
336
337Workflows can react to these too. A published release triggers `on: release`,
338a `POST /dispatches` triggers `on: repository_dispatch`, and webhooks get
339`release`, `status`, `deployment` and `deployment_status` events. A job with
340`environment:` creates a deployment, gets that environment's secrets and
341variables, and is refused if the environment's branch policy doesn't allow
342the ref. As on GitHub, events caused by `GITHUB_TOKEN` don't start new
343workflow runs, except `repository_dispatch`.
344
345**Not supported yet:** GraphQL (`/api/graphql`), setting secrets with a
346sealed-box `encrypted_value` (send `value` instead), environment protection
347rules (required reviewers, wait timers), forks, team reviewers, and code and
348commit search. Search qualifiers for stars, forks and languages are accepted
349but ignored.
350
351## Developing
352
353```sh
354pnpm install
355pnpm -r build && pnpm -r typecheck && pnpm -r test
356pnpm --filter @diffinery/app db:migrate:local
357pnpm dev # http://localhost:8787, signed in as dev@diffinery.local
358pnpm --filter @diffinery/app e2e # wrangler dev + real git + the runner agent + Chromium
359```
360
361The end-to-end test does the following:
362
3631. Pushes a repository with real `git`.
3642. Runs its CI through the actual runner agent.
3653. Checks secret scanning and branch protection.
3664. Has an agent open a pull request that needs a human approval and a green
367 check, then merges it.
3685. Runs pull request workflows on test merge commits, `pull_request_target`
369 from the base branch, and a merge queue with `merge_group` runs, including a
370 group that fails and leaves the queue.
3716. Clones shallow, then deepens the clone.
3727. Drives the MCP server.
3738. In Chromium with a virtual authenticator, joins from an invite with a new
374 passkey, adds and removes passkeys, signs out and in, and uses an admin
375 sign-in link.
3769. Drives the REST API with the real `@octokit/rest` client: contents, git
377 data, statuses as required checks, releases with asset uploads and
378 downloads, issues, reactions, pagination, search and deployments.
379
380Team preferences for product and design decisions are in
381[docs/PREFERENCES.md](docs/PREFERENCES.md).