Diffinery

codician/diffinerypublic

Open-source, agent-first GitHub and GitHub Actions alternative that runs on your own Cloudflare account: git hosting, pull requests, code review and drop-in GitHub Actions YAML on Workers, Durable Objects, R2 and Containers.

Web-only egress for hosted compute behind EGRESS=web, off by default (#134)

Security register X2. With EGRESS=web, hosted Actions job containers and
agent sandboxes (tasks, reviews, terminals) start with enableInternet
false: Cloudflare then lets only ports 80 and 443 and DNS (to its own
resolvers) leave the container. allowedHosts ["*"] with interceptHttps
forwards HTTP and HTTPS for every host through the Worker
(ContainerProxy, now exported from the entrypoint); SMTP, SSH, IRC and
other ports are refused.

- lib/egress.ts: egressFor() decides per start from the account that
  owns the repository, as quotas do; accounts run by a site
  administrator (quotasFor().exempt) stay open, so Diffinery's own
  Deploy and live tests keep their egress. egressStart() maps a policy
  to container settings.
- runner/containers.ts: RunnerContainer.startJob and
  SandboxContainer.boot apply the policy before starting; a limited one
  is kept in the Durable Object so a sandbox started again after its
  idle sleep stays limited.
- Runner agent: under DIFFINERY_EGRESS=web it trusts the CA Cloudflare
  intercepts HTTPS with (system store, NODE_EXTRA_CA_CERTS,
  REQUESTS_CA_BUNDLE, and its own requests); the Docker wrapper passes
  it to the registry relay.
- Docs: EGRESS in self-hosting configuration, an Egress section in
  Limits, and the hosted runner's network row.

Unset or any other value keeps today's open egress. Self-hosted runners
and imports are unaffected.


Claude-Session: https://claude.ai/code/session_016yhgAu6oKfqes812Yd4qjo

Co-authored-by: Claude <noreply@anthropic.com>
Canberk Demirci authored · GitHub committed signed1 parent 0858d86commit 16676eb1bbe878e37f9518ceb8991a321e94608bBrowse files
12 files changed+241−11
Files (12)
modifiedapps/diffinery/docs/actions/runners.md+1−1View file
@@ -66,7 +66,7 @@
6666 | Architecture | `linux/amd64` (`runner.arch` is `X64`) |
6767 | Operating system | Ubuntu 24.04 (`runner.os` is `Linux`) |
6868 | User | `root` (there is no `sudo` command; you don't need it) |
69-| Network | Outbound internet access; no inbound connections |
69+| Network | Outbound internet access (only HTTP and HTTPS on instances that set [`EGRESS=web`](/docs/reference/limits#egress)); no inbound connections |
7070 | Workspace | `/workspace/<repository name>` (`GITHUB_WORKSPACE`) |
7171 | Temp directory | `/workspace/_temp` (`RUNNER_TEMP`) |
7272 | Tool cache | `/opt/hostedtoolcache` (`RUNNER_TOOL_CACHE`, `AGENT_TOOLSDIRECTORY`) |
modifiedapps/diffinery/docs/reference/limits.md+8−1View file
@@ -1,6 +1,6 @@
11 ---
22 title: Limits
3-description: Every hard limit in Diffinery and where it applies: account quotas, sign-in, tokens, API, search, archives, releases, git, Actions, sandboxes, agent tasks, agent review, packages, imports, email.
3+description: Every hard limit in Diffinery and where it applies: quotas, egress, sign-in, tokens, API, search, archives, releases, git, Actions, sandboxes, agent tasks, agent review, packages, imports, email.
44 ---
55
66 This page lists the fixed limits built into Diffinery: sizes, counts, timeouts and lifetimes. Use it when a push, a workflow, an upload or an API call fails with a size or count error, or when you plan something large. Values come from the code of the running version. Where a limit comes from Cloudflare rather than from Diffinery, the table says so.
@@ -181,6 +181,13 @@
181181 | Agent sandboxes open at once | 5 | Across all the account's repositories, including agent tasks', reviews' and terminals' sandboxes. A new one is refused with `429 this account already has 5 agent sandboxes open; close one or wait for one to finish`. |
182182 | Repositories | 500 | Creating, importing or transferring in a repository past it is refused with `403 <account> has reached its limit of 500 repositories; delete one or ask a site administrator to raise it`. |
183183
184+## Egress
185+
186+| Limit | Value | Applies to |
187+| --- | --- | --- |
188+| Network access of hosted compute | The whole internet by default; HTTP and HTTPS (ports 80 and 443) and DNS with `EGRESS=web` | Hosted Actions jobs and agent sandboxes, including tasks, reviews and terminals. Under `web` a connection to any other port fails: SSH (`git@...` remotes), SMTP (25, 465, 587), IRC and the like. The account that owns the repository decides, and accounts run by a site administrator stay open. Self-hosted runners aren't limited. See [EGRESS](/docs/self-hosting/configuration#egress). |
189+| HTTPS under `EGRESS=web` | Intercepted, re-signed with a Cloudflare CA | The runner agent makes the system store, Node.js and Python's `requests` trust the CA. Tools with their own CA list, certificate pinning, client certificates and HTTPS from Docker containers a job starts fail their certificate checks. |
190+
184191 ## Actions
185192
186193 | Limit | Value | Applies to |
modifiedapps/diffinery/docs/self-hosting/configuration.md+17−0View file
@@ -37,6 +37,7 @@
3737 | `IMPORT_GIT_AGENT` | Not set in production; `external` in the `dev` environment | Never in production | With `external`, an import's Git transfer doesn't start a container: it waits for an import agent started by hand (see [The dev environment](#the-dev-environment)). |
3838 | `SCHEDULE_TICK_MINUTES` | Not set | No | Minutes between the workflow scheduler's checks, a whole number from 1 to 60. Default `5`, GitHub's shortest schedule interval. Anything else counts as not set. The end-to-end tests use `1`. |
3939 | `WAIT_TIMER_MINUTE_MS` | Not set | Never in production | Local development and the end-to-end tests only: how long one minute of a deployment environment's wait timer lasts, in milliseconds, from 1 to `60000` (the default). Anything else counts as not set. It counts only when `PUBLIC_URL` is on this machine (`localhost`, `127.x` or `[::1]`) and `ACCESS_AUD` is empty, as for `DEV_AUTH`; a deployed Worker ignores it. The end-to-end tests use `10000` so a wait timer passes in seconds. |
40+| `EGRESS` | Not set | No | `web` limits hosted Actions jobs and agent sandboxes (including tasks, reviews and terminals) to HTTP and HTTPS (ports 80 and 443) and DNS. Any other value, or none, leaves them open to the whole internet. Accounts run by a site administrator stay open. See [EGRESS](#egress). |
4041 | `TERMINALS` | Not set | No | `off` switches [terminals](/docs/agents/terminal) off: nobody can open one, and open ones end at their next check, within a minute. Anything else, or not set, leaves them on. |
4142
4243 When either `ACCESS_TEAM_DOMAIN` or `ACCESS_AUD` is empty, email sign-in at `/auth/access` is off: the page answers "Account recovery by email is not set up on this instance. Ask an administrator for a sign-in link.", and the sign-in page doesn't offer **Recover your account by email**. Passkeys, invitations and admin sign-in links keep working, but the first administrator can't sign in, because email sign-in is how an account in `BOOTSTRAP_ADMINS` is created.
@@ -74,6 +75,22 @@
7475
7576 The limits on codes (5 a day per address, 20 a day per IP address, 10 minutes and 5 tries each) are in [Limits](/docs/reference/limits#sign-in-and-sessions). Sign-up emails always send: they need the `EMAIL` binding and the `MAIL` queue like every other email.
7677
78+### EGRESS
79+
80+**Experimental:** `web` hasn't been tried on a live instance yet, so turn it on for a test account first and run the checks below.
81+
82+With `EGRESS=web`, hosted job and sandbox containers start without direct internet access. Cloudflare then lets only ports 80 and 443, and DNS to Cloudflare's resolvers, leave the container, and Diffinery forwards HTTP and HTTPS requests to any host through the Worker. Unset, or any other value, hosted compute reaches the whole internet as before. The setting is read when a container starts: jobs and sandboxes already running keep the access they started with.
83+
84+Under `web`:
85+
86+- **Works:** `git clone https://...`, `curl`, `apt-get`, `npm`, `pnpm` and `pip` installs, `docker pull`, and calls to the Diffinery and GitHub APIs.
87+- **Refused:** every other port. That includes SSH (`git@...` remotes, `ssh`, `scp`), SMTP (25, 465 and 587), IRC, databases and HTTP on other ports.
88+- **HTTPS is intercepted.** Requests leave through the instance's own Worker, which sees them decrypted, as the operator already can for the container itself. Cloudflare re-signs them with a CA it puts in the container at `/etc/cloudflare/certs/cloudflare-containers-ca.crt`. The runner agent adds that CA to the system store and sets `NODE_EXTRA_CA_CERTS` and `REQUESTS_CA_BUNDLE`, so tools that use the system store, Node.js and Python's `requests` trust it. Tools with their own fixed CA list or certificate pinning, client certificates (mutual TLS), and HTTPS from inside Docker containers a job starts (`docker run`, `services:`, `RUN` steps of a build) fail their certificate checks.
89+
90+The account that owns the repository decides, as for [quotas](/docs/reference/limits#account-quotas). Accounts run by a site administrator (their own account, and organizations with a site administrator as an owner) stay open even with `EGRESS=web`, so the instance's own Deploy and live test workflows keep their access. Self-hosted runners and GitHub imports are not affected. Under `wrangler dev` sandboxes run in a local process, not containers, so the setting changes nothing there.
91+
92+To turn it on, add `"EGRESS": "web"` under `vars` in `wrangler.jsonc`; the deploy keeps it.
93+
7794 ### DEV_AUTH
7895
7996 When `DEV_AUTH` is set, `ACCESS_AUD` is empty and both `PUBLIC_URL` and the request's own address are on the local machine (`localhost`, `127.x.x.x` or `[::1]`), every request without credentials (no token, no session) acts as the account with the email in `DEV_AUTH`, with full access. That account is created on first use and is a site administrator if the email is also in `BOOTSTRAP_ADMINS`. While `DEV_AUTH` is set, the email log also keeps each whole message, and `GET /api/v3/diffinery/emails` returns it, so tests can read emails without delivering them.
modifiedapps/diffinery/src/env.ts+2−0View file
@@ -53,6 +53,8 @@
5353 QUOTA_HOSTED_JOBS?: string;
5454 QUOTA_SANDBOXES?: string;
5555 QUOTA_REPOSITORIES?: string;
56+ /** "web": hosted jobs and sandboxes reach only HTTP, HTTPS and DNS (lib/egress.ts). Anything else, or unset, is open. */
57+ EGRESS?: string;
5658 /** Largest source repository an import will take, in KB (GitHub `size`); default 10 GB. */
5759 IMPORT_MAX_REPO_KB?: string;
5860 /** Optional GitHub OAuth app, for linking an account through the web flow instead of a gist. */
modifiedapps/diffinery/src/index.ts+2−0View file
@@ -12,6 +12,8 @@
1212 export { ImportWorkflow } from "./imports/workflow.js";
1313 export { ImportContainer } from "./imports/container.js";
1414 export { ProjectDO } from "./agentic/ProjectDO.js";
15+// Carries hosted containers' HTTP and HTTPS out under EGRESS=web (lib/egress.ts).
16+export { ContainerProxy } from "@cloudflare/containers";
1517
1618 // The web UI is mounted last so API, git and runner routes take precedence;
1719 // unknown API paths answer in JSON rather than falling through to its HTML 404.
addedapps/diffinery/src/lib/egress.ts+30−0View file
@@ -1,0 +1,30 @@
1+import type { Env } from "../env.js";
2+import { quotasFor } from "./quotas.js";
3+
4+/**
5+ * How much of the internet hosted compute (Actions jobs on hosted runners, agent sandboxes and web
6+ * terminals) reaches, set by EGRESS:
7+ *
8+ * - open (the default): everything.
9+ * - web: HTTP and HTTPS (ports 80 and 443) and DNS only. The container starts without internet
10+ * access, which leaves the platform only those ports and DNS to Cloudflare's resolvers; requests to
11+ * every host (allowedHosts "*") then go out through the Worker (ContainerProxy, exported in
12+ * index.ts). HTTPS is intercepted with a CA the platform puts in the container, which the runner
13+ * agent trusts at start (containers/runner/src/egress.ts). SMTP, SSH and other ports are refused.
14+ *
15+ * As with quotas, the account that owns the repository decides, and accounts run by a site
16+ * administrator stay open, so the instance's own Deploy and live tests keep their egress.
17+ * Self-hosted runners are not affected.
18+ */
19+export type Egress = "open" | "web";
20+
21+export async function egressFor(env: Env, accountId: string): Promise<Egress> {
22+ if (env.EGRESS !== "web") return "open";
23+ return (await quotasFor(env, accountId)).exempt ? "open" : "web";
24+}
25+
26+/** What a hosted container is started with under a policy (runner/containers.ts). */
27+export function egressStart(egress: Egress): { enableInternet: boolean; interceptHttps: boolean; allowedHosts?: string[]; envVars: Record<string, string> } {
28+ if (egress === "open") return { enableInternet: true, interceptHttps: false, envVars: {} };
29+ return { enableInternet: false, interceptHttps: true, allowedHosts: ["*"], envVars: { DIFFINERY_EGRESS: "web" } };
30+}
modifiedapps/diffinery/src/repo/actions.ts+3−2View file
@@ -23,6 +23,7 @@
2323 import { ghRepo, numericId } from "../http/gh.js";
2424 import { HttpError, newId, now, randomToken, sha256 } from "../lib/util.js";
2525 import { localOnly } from "../auth/dev.js";
26+import { egressFor } from "../lib/egress.js";
2627 import { endCompute, mayStart, startCompute } from "../lib/quotas.js";
2728 import { blockedBy, killSwitches, type Work } from "../lib/killswitch.js";
2829 import type { Env } from "../env.js";
@@ -1050,7 +1051,7 @@
10501051 this.repo.upsertCheck({ sha: run.head_sha ?? run.sha, name: this.checkName(run, job), status: "queued", runId: run.id, jobId, detailsUrl: `${this.env.PUBLIC_URL}/${m.owner}/${m.name}/actions/runs/${run.id}/job/${jobId}` });
10511052 try {
10521053 const stub = this.env.RUNNER.get(this.env.RUNNER.idFromName(container));
1053- await stub.startJob({ repoId: m.repoId, jobId, token, apiUrl: this.env.PUBLIC_URL });
1054+ await stub.startJob({ repoId: m.repoId, jobId, token, apiUrl: this.env.PUBLIC_URL, egress: await egressFor(this.env, m.ownerId) });
10541055 } catch (e) {
10551056 const message = (e as Error).message;
10561057 if (message.includes(NO_INSTANCE)) return this.placementFailed(jobId, message);
@@ -2101,7 +2102,7 @@
21012102 GIT_AUTHOR_EMAIL: `${caller.login}@users.noreply.diffinery.com`,
21022103 GIT_COMMITTER_NAME: caller.login,
21032104 GIT_COMMITTER_EMAIL: `${caller.login}@users.noreply.diffinery.com`,
2104- }, secret, id, { id, accountId: m.ownerId, repoId: m.repoId });
2105+ }, secret, id, { id, accountId: m.ownerId, repoId: m.repoId }, await egressFor(this.env, m.ownerId));
21052106 } catch (e) {
21062107 console.error("sandbox boot failed", e);
21072108 this.sql.exec("UPDATE sandboxes SET status = 'failed' WHERE id = ?", id);
modifiedapps/diffinery/src/runner/containers.ts+39−6View file
@@ -2,25 +2,58 @@
22 import { Container } from "@cloudflare/containers";
33 import type { StopParams } from "@cloudflare/containers";
44 import type { Env } from "../env.js";
5+import { egressStart, type Egress } from "../lib/egress.js";
56 import { endCompute, startCompute } from "../lib/quotas.js";
67 import { blockedBy, killSwitches } from "../lib/killswitch.js";
78
89 /**
10+ * Hosted compute (jobs and sandboxes) under the instance's egress policy (lib/egress.ts), set before
11+ * each start. A limited one is kept in storage, so the container starting again from this Durable
12+ * Object with its defaults (a sandbox after its idle sleep) is limited too; the allowed hosts are
13+ * kept by the library itself.
14+ */
15+abstract class HostedContainer extends Container<Env> {
16+ override enableInternet = true;
17+
18+ constructor(ctx: DurableObject["ctx"], env: Env) {
19+ super(ctx, env);
20+ const kept = ctx.storage.kv.get<Egress>("egress");
21+ if (kept) this.adopt(kept);
22+ }
23+
24+ /** Applies the policy before a start and returns the variables that tell the agent about it. */
25+ protected async useEgress(egress: Egress): Promise<Record<string, string>> {
26+ const o = this.adopt(egress);
27+ if (o.allowedHosts) await this.setAllowedHosts(o.allowedHosts);
28+ if (egress !== "open") this.ctx.storage.kv.put("egress", egress);
29+ return o.envVars;
30+ }
31+
32+ private adopt(egress: Egress) {
33+ const o = egressStart(egress);
34+ this.enableInternet = o.enableInternet;
35+ this.interceptHttps = o.interceptHttps;
36+ return o;
37+ }
38+}
39+
40+/**
941 * One container per workflow job. The runner agent inside fetches its job spec
1042 * from the Diffinery API, runs the steps and reports back, then exits.
1143 */
12-export class RunnerContainer extends Container<Env> {
44+export class RunnerContainer extends HostedContainer {
1345 // Jobs make no inbound requests, so activity-based sleep must not stop them.
1446 override sleepAfter = "6h";
15- override enableInternet = true;
1647
17- async startJob(input: { repoId: string; jobId: string; token: string; apiUrl: string }): Promise<void> {
48+ async startJob(input: { repoId: string; jobId: string; token: string; apiUrl: string; egress?: Egress }): Promise<void> {
1849 await this.ctx.storage.put("job", { repoId: input.repoId, jobId: input.jobId });
50+ const egressEnv = await this.useEgress(input.egress ?? "open");
1951 await this.start({
2052 envVars: {
2153 DIFFINERY_API_URL: input.apiUrl,
2254 DIFFINERY_JOB_ID: input.jobId,
2355 DIFFINERY_JOB_TOKEN: input.token,
56+ ...egressEnv,
2457 },
2558 });
2659 }
@@ -48,15 +81,15 @@
4881 * Long-lived task sandbox for AI agents: the repo checked out with toolchains,
4982 * and an HTTP API (exec, read/write files) on port 8080.
5083 */
51-export class SandboxContainer extends Container<Env> {
84+export class SandboxContainer extends HostedContainer {
5285 override defaultPort = 8080;
5386 override sleepAfter = "30m";
54- override enableInternet = true;
5587
56- async boot(envVars: Record<string, string>, secret: string, _name?: string, usage?: { id: string; accountId: string; repoId: string }): Promise<void> {
88+ async boot(envVars: Record<string, string>, secret: string, _name?: string, usage?: { id: string; accountId: string; repoId: string }, egress: Egress = "open"): Promise<void> {
5789 await this.ctx.storage.put("secret", secret);
5890 // Its running time counts against the repository owner's hosted minutes (lib/quotas.ts).
5991 if (usage) await this.ctx.storage.put("usage", usage);
92+ envVars = { ...envVars, ...(await this.useEgress(egress)) };
6093 // The first start on a host after a deploy pulls the runner image, which takes
6194 // well over the library's 8 s / 20 s defaults.
6295 await this.startAndWaitForPorts({
addedapps/diffinery/test/unit/egress.test.ts+100−0View file
@@ -1,0 +1,100 @@
1+import { describe, expect, it } from "vitest";
2+import { DatabaseSync } from "node:sqlite";
3+import { egressFor, egressStart, type Egress } from "../../src/lib/egress.js";
4+import { Actions } from "../../src/repo/actions.js";
5+import { REPO_SCHEMA, migrateRepoSchema } from "../../src/repo/schema.js";
6+import type { RepoDO } from "../../src/repo/RepoDO.js";
7+import type { Caller } from "../../src/repo/types.js";
8+import type { Env } from "../../src/env.js";
9+import { testD1 } from "./stubs/d1.js";
10+
11+const at = "2026-10-01T00:00:00.000Z";
12+
13+/** Accounts as in quotas.test.ts: ann and her organization acme, root (a site administrator) and his organization ops. */
14+function accounts() {
15+ const DB = testD1();
16+ const add = (id: string, kind: string, siteAdmin = 0) => DB.sqlite.prepare("INSERT INTO accounts (id, login, kind, status, site_admin, created_at) VALUES (?, ?, ?, 'active', ?, ?)").run(id, id, kind, siteAdmin, at);
17+ add("ann", "user");
18+ add("root", "user", 1);
19+ add("acme", "org");
20+ add("ops", "org");
21+ DB.sqlite.prepare("INSERT INTO org_members (org_id, user_id, role, created_at) VALUES ('acme', 'ann', 'owner', ?), ('ops', 'root', 'owner', ?)").run(at, at);
22+ return DB;
23+}
24+
25+function sqlStorage() {
26+ const db = new DatabaseSync(":memory:");
27+ return {
28+ exec(query: string, ...args: unknown[]) {
29+ const params = args.map((a) => (a === undefined ? null : a)) as Array<string | number | null>;
30+ let rows: unknown[] = [];
31+ if (!params.length && query.trim().replace(/;\s*$/, "").includes(";")) db.exec(query);
32+ else rows = db.prepare(query).all(...params);
33+ return { toArray: () => rows, one: () => rows[0] };
34+ },
35+ };
36+}
37+
38+/** A repository owned by `owner` whose hosted job launch and sandbox boot record the egress they're given. */
39+function repository(owner: string, vars: Record<string, string>) {
40+ const sql = sqlStorage();
41+ sql.exec(REPO_SCHEMA);
42+ migrateRepoSchema(sql as unknown as SqlStorage);
43+ const started: { job?: Egress; sandbox?: Egress } = {};
44+ const repo = {
45+ sql,
46+ meta: { repoId: "r", owner, ownerId: owner, name: "app", defaultBranch: "main" },
47+ upsertCheck: () => {},
48+ broadcast: () => {},
49+ resolve: async () => "abc",
50+ } as unknown as RepoDO;
51+ const env = {
52+ DB: accounts(),
53+ PUBLIC_URL: "https://d.test",
54+ ...vars,
55+ RUNNER: { idFromName: (n: string) => n, get: () => ({ startJob: async (i: { egress?: Egress }) => void (started.job = i.egress) }) },
56+ SANDBOX: { idFromName: (n: string) => n, get: () => ({ boot: async (...a: unknown[]) => void (started.sandbox = a[4] as Egress) }) },
57+ } as unknown as Env;
58+ const actions = new Actions(repo, env);
59+ const t = new Date().toISOString();
60+ sql.exec("INSERT INTO runs (id, run_number, workflow_path, workflow_name, event, ref, sha, actor, status, created_at, updated_at, event_payload) VALUES ('run1', 1, 'w.yml', 'w', 'push', 'refs/heads/main', 'abc', '{}', 'in_progress', ?, ?, '{}')", t, t);
61+ sql.exec("INSERT INTO jobs (id, run_id, job_key, name, status) VALUES ('j1', 'run1', 'k', 'k', 'queued')");
62+ const caller = { id: "ann", login: "ann", kind: "user", role: "write" } as Caller;
63+ return {
64+ started,
65+ async run() {
66+ await (actions as unknown as { launch(id: string): Promise<void> }).launch("j1");
67+ await actions.createSandbox(caller);
68+ return started;
69+ },
70+ };
71+}
72+
73+describe("egress of hosted compute", () => {
74+ it("is open unless EGRESS is web, and a site administrator's accounts stay open", async () => {
75+ const DB = accounts();
76+ const env = (vars: Record<string, string> = {}) => ({ DB, ...vars }) as unknown as Env;
77+ for (const id of ["ann", "acme", "root", "ops"]) {
78+ expect(await egressFor(env(), id)).toBe("open");
79+ expect(await egressFor(env({ EGRESS: "open" }), id)).toBe("open");
80+ expect(await egressFor(env({ EGRESS: "everything" }), id)).toBe("open");
81+ }
82+ expect(await egressFor(env({ EGRESS: "web" }), "ann")).toBe("web");
83+ expect(await egressFor(env({ EGRESS: "web" }), "acme")).toBe("web");
84+ expect(await egressFor(env({ EGRESS: "web" }), "root")).toBe("open");
85+ expect(await egressFor(env({ EGRESS: "web" }), "ops")).toBe("open");
86+ });
87+
88+ it("web starts a container without internet, HTTP and HTTPS to every host through the Worker", () => {
89+ expect(egressStart("open")).toEqual({ enableInternet: true, interceptHttps: false, envVars: {} });
90+ expect(egressStart("web")).toEqual({ enableInternet: false, interceptHttps: true, allowedHosts: ["*"], envVars: { DIFFINERY_EGRESS: "web" } });
91+ });
92+
93+ it("the repository owner's policy reaches its hosted jobs and sandboxes", async () => {
94+ expect(await repository("acme", {}).run()).toEqual({ job: "open", sandbox: "open" });
95+ expect(await repository("acme", { EGRESS: "web" }).run()).toEqual({ job: "web", sandbox: "web" });
96+ expect(await repository("ann", { EGRESS: "web" }).run()).toEqual({ job: "web", sandbox: "web" });
97+ // Diffinery's own pipelines run in a site administrator's organization: they keep their egress.
98+ expect(await repository("ops", { EGRESS: "web" }).run()).toEqual({ job: "open", sandbox: "open" });
99+ });
100+});
modifiedcontainers/runner/image/docker+2−1View file
@@ -29,7 +29,8 @@
2929 -addext "subjectAltName=DNS:$host" -keyout "$state/relay.key" -out "$state/relay.crt" >/dev/null 2>&1 || return 1
3030 cp "$state/relay.crt" "/etc/docker/certs.d/$host/ca.crt"
3131 printf '[registry."%s"]\n ca = ["%s"]\n' "$host" "$state/relay.crt" >"$state/buildkitd.toml"
32- env -i PATH="$sys_path" DIFFINERY_RELAY_HOST="$host" DIFFINERY_RELAY_UPSTREAM="https://$host" DIFFINERY_RELAY_PORT="$relay_port" \
32+ # Under web-only egress the relay's own HTTPS needs the egress CA (src/egress.ts).
33+ env -i PATH="$sys_path" ${NODE_EXTRA_CA_CERTS:+"NODE_EXTRA_CA_CERTS=$NODE_EXTRA_CA_CERTS"} DIFFINERY_RELAY_HOST="$host" DIFFINERY_RELAY_UPSTREAM="https://$host" DIFFINERY_RELAY_PORT="$relay_port" \
3334 DIFFINERY_RELAY_CERT="$state/relay.crt" DIFFINERY_RELAY_KEY="$state/relay.key" \
3435 setsid -f node /opt/diffinery/agent.mjs --registry-relay >/var/log/diffinery-relay.log 2>&1 </dev/null
3536 for _ in $(seq 1 80); do
addedcontainers/runner/src/egress.ts+35−0View file
@@ -1,0 +1,35 @@
1+import { execFile } from "node:child_process";
2+import { copyFile, readFile } from "node:fs/promises";
3+import tls from "node:tls";
4+import { promisify } from "node:util";
5+
6+/** Where Cloudflare puts the CA it intercepts a container's HTTPS with, once the container is up. */
7+export const EGRESS_CA = "/etc/cloudflare/certs/cloudflare-containers-ca.crt";
8+
9+/**
10+ * Under web-only egress (DIFFINERY_EGRESS=web, apps/diffinery/src/lib/egress.ts) HTTPS leaves the
11+ * container through Cloudflare's interception, signed by EGRESS_CA. The agent trusts it for its own
12+ * requests and adds it to the system store (git, curl, Docker, Python's ssl) and to the variables
13+ * Node.js and Python's requests read, so the steps and commands it starts trust it too.
14+ */
15+export async function trustEgressCa(env: NodeJS.ProcessEnv = process.env): Promise<void> {
16+ if (env.DIFFINERY_EGRESS !== "web") return;
17+ let pem = "";
18+ for (let i = 0; i < 100 && !pem; i++) {
19+ pem = await readFile(EGRESS_CA, "utf8").catch(() => "");
20+ if (!pem) await new Promise((r) => setTimeout(r, 100));
21+ }
22+ if (!pem) {
23+ process.stderr.write(`[diffinery-runner] web-only egress, but ${EGRESS_CA} did not appear: HTTPS will fail\n`);
24+ return;
25+ }
26+ tls.setDefaultCACertificates([...tls.getCACertificates("default"), pem]);
27+ try {
28+ await copyFile(EGRESS_CA, "/usr/local/share/ca-certificates/cloudflare-containers-ca.crt");
29+ await promisify(execFile)("update-ca-certificates");
30+ } catch (e) {
31+ process.stderr.write(`[diffinery-runner] could not add the egress CA to the system store: ${(e as Error).message}\n`);
32+ }
33+ env.NODE_EXTRA_CA_CERTS ??= EGRESS_CA;
34+ env.REQUESTS_CA_BUNDLE ??= "/etc/ssl/certs/ca-certificates.crt";
35+}
modifiedcontainers/runner/src/main.ts+2−0View file
@@ -16,6 +16,7 @@
1616 import { readFile } from "node:fs/promises";
1717 import type { JobSpec } from "@diffinery/shared";
1818 import { apiBaseFor, runAgentJob } from "./agent.js";
19+import { trustEgressCa } from "./egress.js";
1920 import { RunnerApi } from "./api.js";
2021 import { runImportAgent } from "./importer.js";
2122 import { startSandbox } from "./sandbox.js";
@@ -59,6 +60,7 @@
5960 const args = parseArgs(process.argv.slice(2));
6061 const env = process.env;
6162 const workRoot = args.workRoot ?? env.DIFFINERY_WORK_ROOT ?? "/workspace";
63+ await trustEgressCa(env);
6264
6365 if (args.relay) {
6466 const { DIFFINERY_RELAY_HOST: host, DIFFINERY_RELAY_UPSTREAM: upstream, DIFFINERY_RELAY_CERT: cert, DIFFINERY_RELAY_KEY: key } = env;