README.md at b3a19be36ded66f167f82a64c9cf416abce4cc71 · codician/diffinery · Diffinery

1 # Diffinery
2
3 A git forge for teams of people and AI agents, running entirely on Cloudflare.
4 Git hosting, pull requests and reviews, issues, GitHub-Actions-compatible CI,
5 an MCP server for agents, and sandboxes, with no servers to run.
6
7 - **Drop-in for GitHub workflows.** `.github/workflows/*.yml` runs as-is: same
8   triggers, expressions, matrices, `needs`, outputs, caches, artifacts, secrets,
9   OIDC, `actions/checkout`, `setup-node/python/go`, and remote JavaScript and
10   composite actions.
11 - **Built for agents.** Agents are first-class accounts owned by a person or
12   team, with their own tokens and limits. They can't push or merge to protected
13   branches, and their pull requests always need a human approval. Every commit
14   they make is stamped with who they act for. They work through MCP
15   (`/mcp`) or the REST API, and can start sandboxes to build and test.
16 - **Secure by default.** Cloudflare Access handles sign-in, while invites,
17   roles and branch protection decide what people can do. Pushes are scanned for
18   leaked credentials. Secrets use envelope encryption, and every sensitive
19   action goes to an append-only audit log.
20
21 ## How it fits on Cloudflare
22
23 | Piece | Cloudflare product |
24 |---|---|
25 | HTTP, git smart HTTP, REST API, MCP, web UI | Workers (Hono) |
26 | One repository: refs, objects, PRs, issues, reviews, checks, runs | Durable Object with SQLite (`RepoDO`), one per repository |
27 | Large objects, Git LFS, logs, caches, artifacts, event archive | R2 |
28 | Accounts, orgs, tokens, invites, notifications, audit log | D1 |
29 | Webhooks, notifications, audit fan-out | Queues (with a dead-letter queue) |
30 | Workflow runs (durable, survive restarts, 6 h job timeout) | Workflows |
31 | Job runners and agent sandboxes | Containers (Ubuntu 24.04 image in `containers/runner`) |
32 | Scheduled workflows (`on: schedule`) | Cron Triggers |
33 | Live run logs and PR updates | Durable Object WebSockets (hibernation) |
34 | Sign-in | Cloudflare Access (GitHub, Google, email one-time PIN, etc.) |
35
36 ```
37 apps/diffinery        the Worker (src/index.ts), D1 migrations, wrangler config, e2e test
38 apps/cli              the `diffinery` command-line client
39 containers/runner     runner agent + sandbox server, and the container image
40 packages/git-core     git objects, packfiles, deltas, diff/merge (pure TypeScript)
41 packages/workflow-yaml  GitHub Actions workflow parser, expressions, matrix, cron
42 packages/shared       types shared by the Worker and the runner
43 scripts               bootstrap.sh, ensure-secrets.sh, render-config.mjs
44 ```
45
46 ## Deploying
47
48 Deploys run from GitHub Actions on every push to `main` (`.github/workflows/deploy.yml`).
49 The workflow needs two repository secrets:
50
51 - `CLOUDFLARE_API_TOKEN`: account permissions Workers Scripts, D1, R2,
52   Queues, Workflows, Containers and Access: Apps and Policies (all Edit), and
53   zone permission Workers Routes Edit on `diffinery.com`.
54 - `CLOUDFLARE_ACCOUNT_ID`
55
56 On each deploy, `scripts/bootstrap.sh` creates anything that is missing: the D1
57 database, R2 bucket, queues and the two Access applications (sign-in for the
58 site, and a bypass for the git, API, MCP and runner paths, which check
59 Diffinery tokens themselves). It then applies D1 migrations. `wrangler deploy`
60 builds and pushes the runner image. On the first deploy, `ensure-secrets.sh`
61 generates `MASTER_KEY` and the OIDC signing key inside the job, so neither
62 value is ever printed.
63
64 If the token can't read your Zero Trust organization, set a repository
65 **variable** `ACCESS_TEAM_DOMAIN` (for example `codician.cloudflareaccess.com`).
66
67 To let people sign in with GitHub, add GitHub as a login method under Zero
68 Trust → Settings → Authentication. Email one-time PIN works without any setup.
69
70 ## Inviting people
71
72 1. The emails in `BOOTSTRAP_ADMINS` (in `apps/diffinery/wrangler.jsonc`)
73    become site admins the first time they sign in.
74 2. Anyone else who signs in lands on a waiting page. They can request access,
75    which admins approve, or an admin or org owner can send an invite from
76    *Settings → People* or with `diffinery invite friend@example.com --org codician`.
77 3. Repository and org roles (read, triage, write, maintain, admin) decide what
78    each person can do.
79
80 Nobody needs a Cloudflare account. They need an Access login and, for git and
81 the CLI, a personal token from *Settings → Tokens*.
82
83 ```sh
84 git clone https://diffinery.com/codician/website.git   # username: anything, password: your dfy_ token
85 diffinery login && diffinery auth setup-git             # or let the CLI handle credentials
86 ```
87
88 ## Agents
89
90 Create an agent under *Settings → Agents* or with `diffinery agent create reviewer-bot --org codician`.
91 Then point any MCP client at it:
92
93 ```json
94 { "mcpServers": { "diffinery": { "type": "http", "url": "https://diffinery.com/mcp", "headers": { "Authorization": "Bearer dfy_…" } } } }
95 ```
96
97 Tools include `get_file`, `list_tree`, `commit_files` (no git client needed),
98 `create_branch`, `create_pull_request` (with an `intent` for reviewers),
99 `review_pull_request`, `merge_pull_request` (policy-guarded), issues,
100 `list_workflow_runs` and `get_job_logs`, `rerun_workflow`, and
101 `sandbox_create`/`sandbox_exec` for running builds and tests.
102
103 Agent guardrails apply everywhere: git, REST, MCP and the web UI.
104
105 - An agent's role is capped at write, and never exceeds its owner's role.
106 - Agents can't push to the default branch or protected branches, and can't
107   merge there. Turn this on per branch with `agentsMayPush` or `agentsMayMerge`.
108 - A pull request opened by an agent needs at least one human approval, and
109   approvals from agents don't count toward it.
110 - Agents can't change branch protection or read or set secrets.
111
112 ## Workflows
113
114 Put workflow files in `.github/workflows/` (or `.diffinery/workflows/`).
115 Supported triggers are `push`, `pull_request`, `workflow_dispatch` (with
116 inputs) and `schedule`, with branch, tag and path filters. Supported workflow
117 features:
118
119 - `needs` and job outputs, `if` with `success()`, `failure()`, `always()` and
120   `cancelled()`, `strategy.matrix` with include and exclude, `fail-fast` and
121   `max-parallel`, `concurrency` with `cancel-in-progress`, and `timeout-minutes`.
122 - `continue-on-error`, `defaults.run`, `env` at every level, `$GITHUB_OUTPUT`,
123   `$GITHUB_ENV`, `$GITHUB_PATH`, `$GITHUB_STEP_SUMMARY`, and masking of
124   secrets and `::add-mask::` values.
125 - `secrets.*`, `vars.*`, `GITHUB_TOKEN` (a per-job token), and
126   `permissions: id-token: write` for OIDC (issuer `https://diffinery.com`).
127 - `runs-on: ubuntu-latest` (and any `ubuntu-*` or `linux` label) runs in a
128   Cloudflare Container. `runs-on: [self-hosted, …]` jobs wait for a runner you
129   start with `diffinery runner -R owner/repo --labels gpu,macos`. Use that for
130   macOS, GPUs or private networks.
131
132 **Not supported yet:** reusable workflows (`jobs.<id>.uses`), job
133 `container:` and `services:`, Docker container actions, `actions/github-script`,
134 Windows and macOS hosted runners, `pull_request_target`, `merge_group`, and
135 `deepen-not` / partial clone filters. Pull request runs use the head commit
136 rather than a test merge commit. A job that uses an unsupported feature fails
137 with a clear message instead of misbehaving.
138
139 ## Developing
140
141 ```sh
142 pnpm install
143 pnpm -r build && pnpm -r typecheck && pnpm -r test
144 pnpm --filter @diffinery/app db:migrate:local
145 pnpm dev                                  # http://localhost:8787, signed in as dev@diffinery.local
146 pnpm --filter @diffinery/app e2e          # wrangler dev + real git + the runner agent
147 ```
148
149 The end-to-end test does the following:
150
151 1. Pushes a repository with real `git`.
152 2. Runs its CI through the actual runner agent.
153 3. Checks secret scanning and branch protection.
154 4. Has an agent open a pull request that needs a human approval and a green
155    check, then merges it.
156 5. Clones shallow, then deepens the clone.
157 6. Drives the MCP server.
1	# Diffinery
2
3	A git forge for teams of people and AI agents, running entirely on Cloudflare.
4	Git hosting, pull requests and reviews, issues, GitHub-Actions-compatible CI,
5	an MCP server for agents, and sandboxes, with no servers to run.
6
7	- Drop-in for GitHub workflows. `.github/workflows/*.yml` runs as-is: same
8	triggers, expressions, matrices, `needs`, outputs, caches, artifacts, secrets,
9	OIDC, `actions/checkout`, `setup-node/python/go`, and remote JavaScript and
10	composite actions.
11	- Built for agents. Agents are first-class accounts owned by a person or
12	team, with their own tokens and limits. They can't push or merge to protected
13	branches, and their pull requests always need a human approval. Every commit
14	they make is stamped with who they act for. They work through MCP
15	(`/mcp`) or the REST API, and can start sandboxes to build and test.
16	- Secure by default. Cloudflare Access handles sign-in, while invites,
17	roles and branch protection decide what people can do. Pushes are scanned for
18	leaked credentials. Secrets use envelope encryption, and every sensitive
19	action goes to an append-only audit log.
20
21	## How it fits on Cloudflare
22
23	\| Piece \| Cloudflare product \|
24	\|---\|---\|
25	\| HTTP, git smart HTTP, REST API, MCP, web UI \| Workers (Hono) \|
26	\| One repository: refs, objects, PRs, issues, reviews, checks, runs \| Durable Object with SQLite (`RepoDO`), one per repository \|
27	\| Large objects, Git LFS, logs, caches, artifacts, event archive \| R2 \|
28	\| Accounts, orgs, tokens, invites, notifications, audit log \| D1 \|
29	\| Webhooks, notifications, audit fan-out \| Queues (with a dead-letter queue) \|
30	\| Workflow runs (durable, survive restarts, 6 h job timeout) \| Workflows \|
31	\| Job runners and agent sandboxes \| Containers (Ubuntu 24.04 image in `containers/runner`) \|
32	\| Scheduled workflows (`on: schedule`) \| Cron Triggers \|
33	\| Live run logs and PR updates \| Durable Object WebSockets (hibernation) \|
34	\| Sign-in \| Cloudflare Access (GitHub, Google, email one-time PIN, etc.) \|
35
36	```
37	apps/diffinery the Worker (src/index.ts), D1 migrations, wrangler config, e2e test
38	apps/cli the `diffinery` command-line client
39	containers/runner runner agent + sandbox server, and the container image
40	packages/git-core git objects, packfiles, deltas, diff/merge (pure TypeScript)
41	packages/workflow-yaml GitHub Actions workflow parser, expressions, matrix, cron
42	packages/shared types shared by the Worker and the runner
43	scripts bootstrap.sh, ensure-secrets.sh, render-config.mjs
44	```
45
46	## Deploying
47
48	Deploys run from GitHub Actions on every push to `main` (`.github/workflows/deploy.yml`).
49	The workflow needs two repository secrets:
50
51	- `CLOUDFLARE_API_TOKEN`: account permissions Workers Scripts, D1, R2,
52	Queues, Workflows, Containers and Access: Apps and Policies (all Edit), and
53	zone permission Workers Routes Edit on `diffinery.com`.
54	- `CLOUDFLARE_ACCOUNT_ID`
55
56	On each deploy, `scripts/bootstrap.sh` creates anything that is missing: the D1
57	database, R2 bucket, queues and the two Access applications (sign-in for the
58	site, and a bypass for the git, API, MCP and runner paths, which check
59	Diffinery tokens themselves). It then applies D1 migrations. `wrangler deploy`
60	builds and pushes the runner image. On the first deploy, `ensure-secrets.sh`
61	generates `MASTER_KEY` and the OIDC signing key inside the job, so neither
62	value is ever printed.
63
64	If the token can't read your Zero Trust organization, set a repository
65	variable `ACCESS_TEAM_DOMAIN` (for example `codician.cloudflareaccess.com`).
66
67	To let people sign in with GitHub, add GitHub as a login method under Zero
68	Trust → Settings → Authentication. Email one-time PIN works without any setup.
69
70	## Inviting people
71
72	1. The emails in `BOOTSTRAP_ADMINS` (in `apps/diffinery/wrangler.jsonc`)
73	become site admins the first time they sign in.
74	2. Anyone else who signs in lands on a waiting page. They can request access,
75	which admins approve, or an admin or org owner can send an invite from
76	Settings → People or with `diffinery invite friend@example.com --org codician`.
77	3. Repository and org roles (read, triage, write, maintain, admin) decide what
78	each person can do.
79
80	Nobody needs a Cloudflare account. They need an Access login and, for git and
81	the CLI, a personal token from Settings → Tokens.
82
83	```sh
84	git clone https://diffinery.com/codician/website.git # username: anything, password: your dfy_ token
85	diffinery login && diffinery auth setup-git # or let the CLI handle credentials
86	```
87
88	## Agents
89
90	Create an agent under Settings → Agents or with `diffinery agent create reviewer-bot --org codician`.
91	Then point any MCP client at it:
92
93	```json
94	{ "mcpServers": { "diffinery": { "type": "http", "url": "https://diffinery.com/mcp", "headers": { "Authorization": "Bearer dfy_…" } } } }
95	```
96
97	Tools include `get_file`, `list_tree`, `commit_files` (no git client needed),
98	`create_branch`, `create_pull_request` (with an `intent` for reviewers),
99	`review_pull_request`, `merge_pull_request` (policy-guarded), issues,
100	`list_workflow_runs` and `get_job_logs`, `rerun_workflow`, and
101	`sandbox_create`/`sandbox_exec` for running builds and tests.
102
103	Agent guardrails apply everywhere: git, REST, MCP and the web UI.
104
105	- An agent's role is capped at write, and never exceeds its owner's role.
106	- Agents can't push to the default branch or protected branches, and can't
107	merge there. Turn this on per branch with `agentsMayPush` or `agentsMayMerge`.
108	- A pull request opened by an agent needs at least one human approval, and
109	approvals from agents don't count toward it.
110	- Agents can't change branch protection or read or set secrets.
111
112	## Workflows
113
114	Put workflow files in `.github/workflows/` (or `.diffinery/workflows/`).
115	Supported triggers are `push`, `pull_request`, `workflow_dispatch` (with
116	inputs) and `schedule`, with branch, tag and path filters. Supported workflow
117	features:
118
119	- `needs` and job outputs, `if` with `success()`, `failure()`, `always()` and
120	`cancelled()`, `strategy.matrix` with include and exclude, `fail-fast` and
121	`max-parallel`, `concurrency` with `cancel-in-progress`, and `timeout-minutes`.
122	- `continue-on-error`, `defaults.run`, `env` at every level, `$GITHUB_OUTPUT`,
123	`$GITHUB_ENV`, `$GITHUB_PATH`, `$GITHUB_STEP_SUMMARY`, and masking of
124	secrets and `::add-mask::` values.
125	- `secrets.`, `vars.`, `GITHUB_TOKEN` (a per-job token), and
126	`permissions: id-token: write` for OIDC (issuer `https://diffinery.com`).
127	- `runs-on: ubuntu-latest` (and any `ubuntu-*` or `linux` label) runs in a
128	Cloudflare Container. `runs-on: [self-hosted, …]` jobs wait for a runner you
129	start with `diffinery runner -R owner/repo --labels gpu,macos`. Use that for
130	macOS, GPUs or private networks.
131
132	Not supported yet: reusable workflows (`jobs.<id>.uses`), job
133	`container:` and `services:`, Docker container actions, `actions/github-script`,
134	Windows and macOS hosted runners, `pull_request_target`, `merge_group`, and
135	`deepen-not` / partial clone filters. Pull request runs use the head commit
136	rather than a test merge commit. A job that uses an unsupported feature fails
137	with a clear message instead of misbehaving.
138
139	## Developing
140
141	```sh
142	pnpm install
143	pnpm -r build && pnpm -r typecheck && pnpm -r test
144	pnpm --filter @diffinery/app db:migrate:local
145	pnpm dev # http://localhost:8787, signed in as dev@diffinery.local
146	pnpm --filter @diffinery/app e2e # wrangler dev + real git + the runner agent
147	```
148
149	The end-to-end test does the following:
150
151	1. Pushes a repository with real `git`.
152	2. Runs its CI through the actual runner agent.
153	3. Checks secret scanning and branch protection.
154	4. Has an agent open a pull request that needs a human approval and a green
155	check, then merges it.
156	5. Clones shallow, then deepens the clone.
157	6. Drives the MCP server.