Diffinery

codician/diffinerypublic

Open-source, agent-first GitHub and GitHub Actions alternative that runs on your own Cloudflare account: git hosting, pull requests, code review and drop-in GitHub Actions YAML on Workers, Durable Objects, R2 and Containers.

Uploads: join large artifacts and caches in Durable Objects of their own, not the repository's (#234)

* Uploads: join large artifacts and caches in Durable Objects of their own, not the repository's
test-bot authored 1 parent b19a33ecommit ac745d753f2cb8d1cbb01ce04b5618980c708f3aBrowse files
9 files changed+35−6
Files (9)
modifiedapps/diffinery/docs/self-hosting/architecture.md+2−1View file
@@ -23,6 +23,7 @@
2323 | Account exports and the clean-up after an account is deleted (durable, with retried steps) | Workflows (`diffinery-account`, class `AccountWorkflow`) | `ACCOUNT_WORKFLOW` |
2424 | The Git transfer of a GitHub import | Containers (`ImportContainer`, one container per transfer) | `IMPORTER` |
2525 | Scheduled workflows | Durable Object alarm, every 5 minutes (`SchedulerDO`, one instance) | `SCHEDULER` |
26+| Joining large artifact and cache uploads | Durable Objects, one per step (`ComposeDO`) | `COMPOSE` |
2627 | Token expiry reminders, email log pruning, registry cleanup, expiry of draft imports and of account exports, a check that the scheduler's alarm is set | Cron Triggers, every 5 minutes | `scheduled` handler |
2728 | Live run logs and pull request updates in the browser | Durable Object WebSockets with hibernation | Through `REPO` |
2829 | Outgoing email | Email Sending | `EMAIL` |
@@ -53,7 +54,7 @@
5354
5455 ## The Worker
5556
56-`apps/diffinery/src/index.ts` exports the Worker's `fetch`, `queue` and `scheduled` handlers, and the classes Cloudflare instantiates: `RepoDO`, `SchedulerDO`, `RunnerContainer`, `SandboxContainer`, `RunWorkflow`, `ImportWorkflow` and `ImportContainer`.
57+`apps/diffinery/src/index.ts` exports the Worker's `fetch`, `queue` and `scheduled` handlers, and the classes Cloudflare instantiates: `RepoDO`, `SchedulerDO`, `ComposeDO`, `RunnerContainer`, `SandboxContainer`, `RunWorkflow`, `ImportWorkflow` and `ImportContainer`.
5758
5859 Every request goes through one middleware before it reaches a route:
5960
modifiedapps/diffinery/docs/self-hosting/configuration.md+2−1View file
@@ -225,13 +225,14 @@
225225 | `RL_AUTH`, `RL_ANON`, `RL_ACCOUNT`, `RL_CREATE`, `RL_MODEL` | Rate limiting (`ratelimits`) | Namespaces `1001` to `1005`; 20, 300, 2,000, 60 and 600 requests per 60 seconds | [Request rate limits](/docs/reference/rest-api#rate-limits): sign-in and security reports per IP address, anonymous requests per IP address, signed-in requests per account, new content per account, and agents' model requests per paying person (`MODEL_RATE_LIMIT` in `src/lib/rate-limit.ts`). Optional: a missing binding (the `dev` environment declares none) or one that fails lets requests through. If you change a limit, change `RATE_LIMITS` (or `MODEL_RATE_LIMIT`) in `src/lib/rate-limit.ts` to match, since the 429 messages quote it. Namespace IDs are shared across your Cloudflare account, so pick others if another Worker already uses these. |
226226 | `REPO` | Durable Object namespace | Class `RepoDO` (SQLite) | One object per repository. |
227227 | `SCHEDULER` | Durable Object namespace | Class `SchedulerDO` (SQLite) | One object that starts scheduled workflow runs on its own alarm. |
228+| `COMPOSE` | Durable Object namespace | Class `ComposeDO` (SQLite) | Joins the blocks of large artifact and cache uploads, one short-lived object per step, so the copying doesn't slow the repository's object. Without it, `RepoDO` does the joining. |
228229 | `RUNNER` | Durable Object namespace with a container | Class `RunnerContainer` (SQLite) | One container per hosted workflow job. |
229230 | `SANDBOX` | Durable Object namespace with a container | Class `SandboxContainer` (SQLite) | One container per agent sandbox. |
230231 | `RUN_WORKFLOW` | Workflow | `diffinery-run`, class `RunWorkflow` | One durable instance per workflow run. |
231232 | `IMPORTER` | Durable Object namespace with a container | Class `ImportContainer` (SQLite) | One container per Git transfer of a GitHub import. |
232233 | `IMPORT_WORKFLOW` | Workflow | `diffinery-import`, class `ImportWorkflow` | One durable instance per attempt of a GitHub import. |
233234
234-The Durable Object classes are declared in four migrations (`new_sqlite_classes`): tag `v1` for `RepoDO`, `RunnerContainer` and `SandboxContainer`, tag `v2` for `ImportContainer`, tag `v3` for `ProjectDO` and tag `v4` for `SchedulerDO`. `wrangler deploy` applies a migration tag the account doesn't have yet.
235+The Durable Object classes are declared in six migrations (`new_sqlite_classes`): tag `v1` for `RepoDO`, `RunnerContainer` and `SandboxContainer`, tag `v2` for `ImportContainer`, tag `v3` for `ProjectDO`, tag `v4` for `SchedulerDO`, tag `v5` for `RunnerContainerMedium`, `RunnerContainerSmall` and `RunnerContainerTiny`, and tag `v6` for `ComposeDO`. `wrangler deploy` applies a migration tag the account doesn't have yet.
235236
236237 ## Other Worker settings
237238
modifiedapps/diffinery/docs/self-hosting/deploy.md+1−1View file
@@ -174,7 +174,7 @@
174174
175175 ### wrangler deploy
176176
177-`pnpm exec wrangler deploy -c wrangler.deploy.jsonc` (run in `apps/diffinery`) deploys the Worker with all its bindings, and builds the runner image from `containers/runner/Dockerfile` (Ubuntu 24.04 with Node.js 22, git, git-lfs and the runner agent) for all three container classes: `RunnerContainer` (hosted jobs), `SandboxContainer` (agent sandboxes) and `ImportContainer` (the Git transfers of GitHub imports). It also applies Durable Object migrations the account doesn't have yet: tag `v1` creates `RepoDO`, `RunnerContainer` and `SandboxContainer`, tag `v2` creates `ImportContainer`, tag `v3` creates `ProjectDO`, and tag `v4` creates `SchedulerDO`.
177+`pnpm exec wrangler deploy -c wrangler.deploy.jsonc` (run in `apps/diffinery`) deploys the Worker with all its bindings, and builds the runner image from `containers/runner/Dockerfile` (Ubuntu 24.04 with Node.js 22, git, git-lfs and the runner agent) for all three container classes: `RunnerContainer` (hosted jobs), `SandboxContainer` (agent sandboxes) and `ImportContainer` (the Git transfers of GitHub imports). It also applies Durable Object migrations the account doesn't have yet: tag `v1` creates `RepoDO`, `RunnerContainer` and `SandboxContainer`, tag `v2` creates `ImportContainer`, tag `v3` creates `ProjectDO`, tag `v4` creates `SchedulerDO`, tag `v5` creates the smaller runner classes, and tag `v6` creates `ComposeDO`.
178178
179179 ### scripts/ensure-secrets.sh
180180
addedapps/diffinery/src/compose.ts+18−0View file
@@ -1,0 +1,18 @@
1+import { DurableObject } from "cloudflare:workers";
2+import type { Env } from "./env.js";
3+import { finalKey, runComposeStep, stagingPrefix, stepKeys, type BlobBucket, type ComposeStep } from "./lib/blob-store.js";
4+import type { BlobRef } from "./lib/runtime-auth.js";
5+
6+/**
7+ * Runs one step of joining an upload's blocks (lib/blob-store.ts `compose`): each step in an object
8+ * of its own (a fresh id), so the steps of a large artifact or cache really run at once and the
9+ * copying doesn't load the repository's object, which serves the rest of its traffic. Ran in RepoDO
10+ * before: eight steps at once made its other calls up to ten times slower while a 200 MiB upload
11+ * was joined. Only the Worker calls it, after checking the upload may still be written.
12+ */
13+export class ComposeDO extends DurableObject<Env> {
14+ async step(b: BlobRef, step: ComposeStep, contentType?: string): Promise<Array<{ partNumber: number; etag: string }>> {
15+ if (!stepKeys(step).every((k) => k === finalKey(b) || k.startsWith(stagingPrefix(b)))) throw new Error("the step names objects outside this upload");
16+ return runComposeStep(this.env.BUCKET as unknown as BlobBucket, step, contentType);
17+ }
18+}
modifiedapps/diffinery/src/env.ts+3−0View file
@@ -1,6 +1,7 @@
11 import type { DiffineryEvent } from "@diffinery/shared";
22 import type { RepoDO } from "./repo/RepoDO.js";
33 import type { SchedulerDO } from "./scheduler.js";
4+import type { ComposeDO } from "./compose.js";
45 import type { ImportContainer } from "./imports/container.js";
56 import type { RunnerContainer, SandboxContainer } from "./runner/containers.js";
67 import type { MailJob } from "./mail/types.js";
@@ -15,6 +16,8 @@
1516 REPO: DurableObjectNamespace<RepoDO>;
1617 /** Fires workflow schedules (src/scheduler.ts). */
1718 SCHEDULER: DurableObjectNamespace<SchedulerDO>;
19+ /** Joins the blocks of large artifact and cache uploads, a step per object (src/compose.ts); without it RepoDO does. */
20+ COMPOSE?: DurableObjectNamespace<ComposeDO>;
1821 /** Minutes between scheduler ticks, 1 to 60 (default 5, GitHub's shortest schedule interval). */
1922 SCHEDULE_TICK_MINUTES?: string;
2023 /** End-to-end tests only: how long one minute of an environment's wait timer lasts, in ms (default 60000). */
modifiedapps/diffinery/src/http/actions-runtime.ts+2−2View file
@@ -67,12 +67,12 @@
6767 return null;
6868 }
6969
70-/** Composes an upload's staged pieces, each step in a Durable Object call of its own (lib/blob-store.ts). */
70+/** Composes an upload's staged pieces, each step in a Durable Object of its own (compose.ts, lib/blob-store.ts). */
7171 function composeUpload(env: Env, b: BlobRef, segments: Array<{ key: string; size: number }>, contentType: string): Promise<number> {
7272 const stub = repoStub(env, b.repo);
7373 return compose(bucket(env), finalKey(b), segments, contentType, {
7474 scratch: stagingPrefix(b),
75- run: (step: ComposeStep) => stub.rtComposeStep({ kind: b.kind, id: b.id }, step, contentType),
75+ run: (step: ComposeStep) => (env.COMPOSE ? env.COMPOSE.get(env.COMPOSE.newUniqueId()).step(b, step, contentType) : stub.rtComposeStep({ kind: b.kind, id: b.id }, step, contentType)),
7676 });
7777 }
7878
modifiedapps/diffinery/src/index.ts+1−0View file
@@ -7,6 +7,7 @@
77
88 export { RepoDO } from "./repo/RepoDO.js";
99 export { SchedulerDO } from "./scheduler.js";
10+export { ComposeDO } from "./compose.js";
1011 export { RunnerContainer, RunnerContainerMedium, RunnerContainerSmall, RunnerContainerTiny, SandboxContainer, SandboxPoolDO } from "./runner/containers.js";
1112 export { RunWorkflow } from "./runner/workflow.js";
1213 export { ImportWorkflow } from "./imports/workflow.js";
modifiedapps/diffinery/test/e2e/entry.ts+1−0View file
@@ -6,6 +6,7 @@
66
77 export { RepoDO } from "../../src/repo/RepoDO.js";
88 export { SchedulerDO } from "../../src/scheduler.js";
9+export { ComposeDO } from "../../src/compose.js";
910 export { RunnerContainer, RunnerContainerMedium, RunnerContainerSmall, RunnerContainerTiny, SandboxContainer } from "../../src/runner/containers.js";
1011 export { RunWorkflow } from "../../src/runner/workflow.js";
1112 export { ImportWorkflow } from "../../src/imports/workflow.js";
modifiedapps/diffinery/wrangler.jsonc+5−1View file
@@ -58,6 +58,7 @@
5858 "bindings": [
5959 { "name": "REPO", "class_name": "RepoDO" },
6060 { "name": "SCHEDULER", "class_name": "SchedulerDO" },
61+ { "name": "COMPOSE", "class_name": "ComposeDO" },
6162 { "name": "RUNNER", "class_name": "RunnerContainer" },
6263 // Smaller hosted runners (src/runner/sizes.ts), the same runner on smaller instance types.
6364 { "name": "RUNNER_MEDIUM", "class_name": "RunnerContainerMedium" },
@@ -76,7 +77,9 @@
7677 // Workflow schedules (scheduler.ts).
7778 { "tag": "v4", "new_sqlite_classes": ["SchedulerDO"] },
7879 // Smaller hosted runner sizes (src/runner/sizes.ts).
79- { "tag": "v5", "new_sqlite_classes": ["RunnerContainerMedium", "RunnerContainerSmall", "RunnerContainerTiny"] }
80+ { "tag": "v5", "new_sqlite_classes": ["RunnerContainerMedium", "RunnerContainerSmall", "RunnerContainerTiny"] },
81+ // Joining large uploads' blocks, a step per object (compose.ts).
82+ { "tag": "v6", "new_sqlite_classes": ["ComposeDO"] }
8083 ],
8184 "workflows": [
8285 { "name": "diffinery-run", "binding": "RUN_WORKFLOW", "class_name": "RunWorkflow" },
@@ -147,6 +150,7 @@
147150 "bindings": [
148151 { "name": "REPO", "class_name": "RepoDO" },
149152 { "name": "SCHEDULER", "class_name": "SchedulerDO" },
153+ { "name": "COMPOSE", "class_name": "ComposeDO" },
150154 { "name": "RUNNER", "class_name": "RunnerContainer" },
151155 { "name": "RUNNER_MEDIUM", "class_name": "RunnerContainerMedium" },
152156 { "name": "RUNNER_SMALL", "class_name": "RunnerContainerSmall" },