Diffinery

codician/diffinerypublic

Open-source, agent-first GitHub and GitHub Actions alternative that runs on your own Cloudflare account: git hosting, pull requests, code review and drop-in GitHub Actions YAML on Workers, Durable Objects, R2 and Containers.

Hosted job limits per account and per instance, not 20 per repository (#222)

* Hosted job limits per account and per instance, not 20 per repository
test-bot authored 1 parent 3acc949commit 87ba4af0d7a8b1ca5697d203e35e83a911caac2cBrowse files
10 files changed+114−40
Files (10)
modifiedapps/diffinery/docs/actions/overview.md+1−1View file
@@ -446,7 +446,7 @@
446446 | `No Diffinery runner matches runs-on: <labels>. Linux (ubuntu-*) runners are hosted; anything else needs runs-on: [self-hosted, ...] and a registered runner.` | Hosted runners accept `ubuntu-*`, `linux`, `diffinery`, `diffinery-*`, `x64` and `amd64`. For other platforms, add `self-hosted` and start a runner with matching labels. See [Runners](/docs/actions/runners). |
447447 | A self-hosted job stays `waiting` | No runner with all the job's labels is polling. Start `diffinery runner` with those labels. After 6 hours the job times out. |
448448 | `The job was not started: this account has used its <n> hosted minutes for this month ...` | The account that owns the repository used its monthly hosted minutes. They renew on the 1st (UTC); until then use a self-hosted runner, or ask a site administrator to raise the quota. See [Account quotas](/docs/reference/limits#account-quotas). |
449-| A hosted job stays queued, then fails with `The job waited 6 hours for one of this account's hosted jobs to finish, and was not started.` | The account already runs as many hosted jobs at once as its quota allows. Queued jobs start as running ones finish. See [Account quotas](/docs/reference/limits#account-quotas). |
449+| A hosted job stays queued, then fails with `The job waited 6 hours for a hosted runner (one of this account's hosted jobs to finish, or a free runner), and was not started.` | The account already runs as many hosted jobs at once as its quota allows (20 by default, at most 40), or every hosted runner of the instance was busy. Queued jobs start as running ones finish. See [Account quotas](/docs/reference/limits#account-quotas) and [hosted runner limits](/docs/reference/limits#actions). |
450450 | `The job was not started: Actions are suspended for this repository by a site administrator: <reason>.` (or `for every repository of <account>`) | A site administrator suspended Actions here. Nothing runs until they lift it, and a red banner on the Actions page shows the reason. See [Kill switches](/docs/admin/site-administration#kill-switches). |
451451 | `The job was not started: Hosted runners are paused on this instance by a site administrator: <reason>. Self-hosted runners still work.` | Hosted compute is paused on the whole instance. Use a self-hosted runner, or re-run once it is resumed. See [Kill switches](/docs/admin/site-administration#kill-switches). |
452452 | `Could not start a runner: ...` | The job's container failed to start with an error that trying again won't fix (passing platform errors are retried first; see [Limits](/docs/reference/limits)). Re-run the run. |
modifiedapps/diffinery/docs/reference/limits.md+4−3View file
@@ -182,7 +182,7 @@
182182 | Quota | Default | What happens past it |
183183 | --- | --- | --- |
184184 | Hosted minutes per month | 3,000 | Hosted Actions jobs and agent sandboxes (including agent tasks, reviews and terminals) add up their running time, rounded up to the minute, per calendar month in UTC. Once it is used up, a hosted job fails without starting ("The job was not started: this account has used its 3000 hosted minutes for this month ..."), and a new sandbox is refused with `429`. Self-hosted runners don't count. |
185-| Hosted jobs at once | 5 | Across all the account's repositories. Further jobs wait as queued and start, oldest first, as running ones finish (checked at least once a minute). A job that waits 6 hours fails with "The job waited 6 hours for one of this account's hosted jobs to finish, and was not started." |
185+| Hosted jobs at once | 20 | Across all the account's repositories, like GitHub's per-account limit (20 on its Free plan). Further jobs wait as queued and start, oldest first, as running ones finish (checked at least once a minute). A job that waits 6 hours fails with "The job waited 6 hours for a hosted runner (one of this account's hosted jobs to finish, or a free runner), and was not started." Accounts run by a site administrator have no quota but are still held to half the fleet (see [Actions](#actions)). |
186186 | Agent sandboxes open at once | 5 | Across all the account's repositories, including agent tasks', reviews' and terminals' sandboxes. A new one is refused with `429 this account already has 5 agent sandboxes open; close one or wait for one to finish`. |
187187 | Repositories | 500 | Creating, importing or transferring in a repository past it is refused with `403 <account> has reached its limit of 500 repositories; delete one or ask a site administrator to raise it`. |
188188 | Storage | 10 GB (10,737,418,240 bytes) | What the account's repositories store (git objects as stored, compressed; Git LFS objects; release assets) and its container images' blobs and manifests (a blob several of its images use counts once). Actions artifacts and caches don't count: they have their own limits per repository (see [Caches and artifacts](#caches-and-artifacts)) and expire. A write that would take the account past it is refused with `<account> has reached its storage limit of 10737418240 bytes (git objects, Git LFS objects, release assets and packages); delete some or ask a site administrator to raise it`: a push that brings new objects (git shows `remote: Push refused: ...` and rejects every ref with `storage limit reached`), Git LFS uploads (each object gets error `507` in the batch response), release asset uploads, files and blobs written through the API, the web editor or MCP (`403`), container image uploads (`403 DENIED`), and starting a GitHub import (`403`); an import that reaches it on the way stops with the same message. Reads, merges, deletes and pushes that only delete or move refs keep working. A deleted branch's objects stay stored until you [purge unreachable objects](/docs/security/push-protection#removing-a-secret-that-was-pushed); deleting releases, assets, image versions or the repository frees their bytes. Repositories made before this quota existed are measured the first time their account's storage is checked. |
@@ -213,8 +213,9 @@
213213 | Schedule catch-up | 60 minutes | A check that comes late, or fails, is caught up by the next one, up to an hour back. Each schedule starts once per check, however many of its minutes the check covers. |
214214 | Starting a job | 3 retries | Starting a job's container is retried 3 times with exponential backoff from 10 seconds before the job fails with "Could not start a runner". |
215215 | Hosted runner sizes | large: `standard-4` (4 vCPU, 12 GiB of memory, 20 GB of disk); medium: `standard-3` (2 vCPU, 8 GiB, 16 GB); small: `standard-1` (1/2 vCPU, 4 GiB, 8 GB); tiny: `basic` (1/4 vCPU, 1 GiB, 4 GB) | Each hosted job runs in its own container. A job runs on large unless `runs-on` names a size or its earlier runs show a smaller one fits with headroom ([Runner sizes](/docs/actions/runners#runner-sizes)). |
216-| Hosted job containers at once | 100 | `max_instances` of the runner container class. |
217-| Hosted jobs at once per repository | 20 | Further jobs stay queued and start, oldest first, as the repository's own jobs finish, so one repository can't take every container. Self-hosted jobs don't count. |
216+| Hosted job containers at once | 100 per size | `max_instances` of each runner container class. |
217+| Hosted jobs at once on the instance | 80 | `HOSTED_JOBS_LIMIT`, kept under the container classes' `max_instances` with room for jobs starting side by side. Past it, every account's next hosted job waits as queued, as for its own quota, until one finishes. Self-hosted jobs don't count. |
218+| Hosted jobs at once per account | Half the instance's, 40 | Whatever the account's quota says, so no account can keep the others' jobs from starting. A repository can use all of its account's share. Further jobs stay queued and start, oldest first, as running ones finish. |
218219 | No free container | Wait, then retry, 8 times | When the platform has no container free for a job (many jobs starting at once, or just after a deploy), the start fails on the platform's side in a way that passes ("Container sidecar is shutting down", a dropped connection), or the job's container stops before its runner fetched the job, the job goes back to queued and starts again after 30 seconds, then 1, 2, 4 and 5 minutes. After 8 tries, about half an hour, it fails with `No hosted runner could be started`. Only jobs whose runner never started are retried, so no step runs twice. |
219220 | Unfinished runs per repository | 500 | Past it, a `workflow_dispatch` or `repository_dispatch` is refused with `429 this repository has 500 unfinished workflow runs, the most allowed; wait for some to finish`, and other events start no run. |
220221 | Self-hosted runners per repository | 100, each with at most 20 labels | Registering one more gets `429 this repository has 100 self-hosted runners, the most allowed; remove one first`. A registration token works for an hour. |
modifiedapps/diffinery/docs/self-hosting/configuration.md+2−1View file
@@ -31,7 +31,8 @@
3131 | `SIGNUP` | Not set | No | `open` lets anyone create an account at `/signup`. See [SIGNUP](#signup). Any other value, or none, keeps the instance invite-only. |
3232 | `TURNSTILE_SITE_KEY` | Set by the deploy | With `SIGNUP=open`; for anonymous security reports | The site key of the Cloudflare Turnstile widget on the sign-up page and the [security report form](#security-contact). The deploy creates the widget and sets this; see [Turnstile](#turnstile). Setting it doesn't open sign-up. |
3333 | `IMPORT_MAX_REPO_KB` | Not set | No | The largest GitHub repository a one-repository import accepts, in KB as GitHub reports a repository's `size`. Default `10485760` (10 GB). An empty value, `0`, a negative number or text that isn't a number counts as not set. A larger source is refused before the import starts with `422 the repository is about N GB, over the M GB import limit; ask an administrator to raise IMPORT_MAX_REPO_KB or split the repository`. See [GitHub imports](#github-imports). |
34-| `QUOTA_HOSTED_MINUTES`, `QUOTA_HOSTED_JOBS`, `QUOTA_SANDBOXES`, `QUOTA_REPOSITORIES`, `QUOTA_STORAGE_BYTES` | Not set | No | The default per-account quotas: hosted minutes per month (default `3000`), hosted Actions jobs at once (`5`), agent sandboxes open at once (`5`), repositories (`500`) and storage in bytes (`10737418240`, 10 GB). `-1` is unlimited; text that isn't a whole number counts as not set. Site administrators change them for one account with `PATCH /api/v3/admin/accounts/{login}/quotas`, and accounts run by a site administrator have none. See [Limits](/docs/reference/limits#account-quotas). |
34+| `QUOTA_HOSTED_MINUTES`, `QUOTA_HOSTED_JOBS`, `QUOTA_SANDBOXES`, `QUOTA_REPOSITORIES`, `QUOTA_STORAGE_BYTES` | Not set | No | The default per-account quotas: hosted minutes per month (default `3000`), hosted Actions jobs at once (`20`), agent sandboxes open at once (`5`), repositories (`500`) and storage in bytes (`10737418240`, 10 GB). `-1` is unlimited; text that isn't a whole number counts as not set. Site administrators change them for one account with `PATCH /api/v3/admin/accounts/{login}/quotas`, and accounts run by a site administrator have none. See [Limits](/docs/reference/limits#account-quotas). |
35+| `HOSTED_JOBS_LIMIT` | Not set | No | Hosted Actions jobs the whole instance runs at once (default `80`); one account runs at most half of them, accounts run by a site administrator included. Keep it under the runner container classes' `max_instances`. Past it, hosted jobs wait as queued until one finishes. See [Limits](/docs/reference/limits#actions). |
3536 | `IMPORT_GHE_HOST` | Not set | No | The host name of a GitHub Enterprise Server (like `ghe.example.com`) that imports may download release assets from. Without it, imports download release assets only from github.com and its asset hosts, and assets of an Enterprise import are skipped with a warning. |
3637 | `GITHUB_OAUTH_CLIENT_ID` | Not set | No | The client ID of a GitHub OAuth app, for linking GitHub accounts with a GitHub sign-in instead of a gist. Needs `GITHUB_OAUTH_CLIENT_SECRET` too. See [GitHub imports](#github-imports). |
3738 | `IMPORT_GIT_AGENT` | Not set in production; `external` in the `dev` environment | Never in production | With `external`, an import's Git transfer doesn't start a container: it waits for an import agent started by hand (see [The dev environment](#the-dev-environment)). |
modifiedapps/diffinery/src/env.ts+2−0View file
@@ -68,6 +68,8 @@
6868 QUOTA_SANDBOXES?: string;
6969 QUOTA_REPOSITORIES?: string;
7070 QUOTA_STORAGE_BYTES?: string;
71+ /** Hosted Actions jobs the whole instance runs at once (lib/quotas.ts fleetLimits); keep it under the runner classes' max_instances. */
72+ HOSTED_JOBS_LIMIT?: string;
7173 /** "web": hosted jobs and sandboxes reach only HTTP, HTTPS and DNS (lib/egress.ts). Anything else, or unset, is open. */
7274 EGRESS?: string;
7375 /** Largest source repository an import will take, in KB (GitHub `size`); default 10 GB. */
modifiedapps/diffinery/src/lib/quotas.ts+44−8View file
@@ -17,12 +17,16 @@
1717 * can override them per account (account_quotas). -1 means unlimited. Accounts run by a site
1818 * administrator (one, or an organization with one as owner) have no quotas, so the instance's own
1919 * work never stops on them.
20+ *
21+ * Over every account, the instance runs at most fleetLimits().total hosted jobs at once, and one account
22+ * at most half of them, exempt accounts included: the fleet stays under the runner container classes'
23+ * max_instances, and no account can keep the others' jobs from starting.
2024 */
2125 export type QuotaKey = "hosted_minutes" | "hosted_jobs" | "sandboxes" | "repositories" | "storage_bytes";
2226 export const QUOTA_KEYS: readonly QuotaKey[] = ["hosted_minutes", "hosted_jobs", "sandboxes", "repositories", "storage_bytes"];
2327 export type Quotas = Record<QuotaKey, number>;
2428
25-export const DEFAULT_QUOTAS: Quotas = { hosted_minutes: 3000, hosted_jobs: 5, sandboxes: 5, repositories: 500, storage_bytes: 10 * 1024 ** 3 };
29+export const DEFAULT_QUOTAS: Quotas = { hosted_minutes: 3000, hosted_jobs: 20, sandboxes: 5, repositories: 500, storage_bytes: 10 * 1024 ** 3 };
2630 const ENV_KEYS: Record<QuotaKey, string> = { hosted_minutes: "QUOTA_HOSTED_MINUTES", hosted_jobs: "QUOTA_HOSTED_JOBS", sandboxes: "QUOTA_SANDBOXES", repositories: "QUOTA_REPOSITORIES", storage_bytes: "QUOTA_STORAGE_BYTES" };
2731 /** The largest quota a site administrator can set: storage is in bytes, the rest are counts. */
2832 const MAX_QUOTA: Record<QuotaKey, number> = { hosted_minutes: 1e9, hosted_jobs: 1e9, sandboxes: 1e9, repositories: 1e9, storage_bytes: 2 ** 50 };
@@ -157,18 +161,50 @@
157161 const within = (limit: number, used: number) => limit < 0 || used < limit;
158162
159163 /**
164+ * Hosted jobs the instance runs at once by default. Under the runner container classes' max_instances
165+ * (100) with room to spare: repositories start jobs side by side, each counting before the others' starts
166+ * are recorded, so a burst can pass the limit by a few.
167+ */
168+export const HOSTED_JOBS_LIMIT = 80;
169+
170+/** Hosted jobs at once: `total` over the whole instance (HOSTED_JOBS_LIMIT), `share` for one account (half of it). */
171+export function fleetLimits(env: Env): { total: number; share: number } {
172+ const n = Math.trunc(Number(env.HOSTED_JOBS_LIMIT));
173+ const total = Number.isFinite(n) && n >= 2 ? n : HOSTED_JOBS_LIMIT;
174+ return { total, share: Math.floor(total / 2) };
175+}
176+
177+/** Hosted jobs running now, the instance's or one account's (a row left open past MAX_RUN_MS no longer counts, as in computeUsage). */
178+async function runningJobs(env: Env, accountId?: string): Promise<number> {
179+ const since = new Date(Date.now() - MAX_RUN_MS.job).toISOString();
180+ const q = accountId
181+ ? env.DB.prepare("SELECT count(*) AS n FROM compute_usage WHERE account_id = ? AND kind = 'job' AND ended_at IS NULL AND started_at > ?").bind(accountId, since)
182+ : env.DB.prepare("SELECT count(*) AS n FROM compute_usage WHERE ended_at IS NULL AND started_at > ? AND kind = 'job'").bind(since);
183+ return (await q.first<{ n: number }>())?.n ?? 0;
184+}
185+
186+/**
160187 * Whether the account may start a hosted job or a sandbox now: "ok", "wait" (a job waits for one of
161188 * the account's running jobs to end), or an error message (minutes used up, or too many sandboxes).
162189 */
163190 export async function mayStart(env: Env, accountId: string, kind: "job" | "sandbox"): Promise<{ ok: true } | { ok: false; wait: boolean; message: string }> {
164191 const q = await quotasFor(env, accountId);
165- if (q.exempt) return { ok: true };
166- // Not usageOf: a start doesn't need the repository count or the storage total (up to 25 repositories asked).
167- const u = await computeUsage(env, accountId);
168- if (!within(q.hosted_minutes, u.hosted_minutes))
169- return { ok: false, wait: false, message: `this account has used its ${q.hosted_minutes} hosted minutes for this month (Actions jobs on hosted runners and agent sandboxes); they renew on the 1st (UTC). Self-hosted runners still work` };
170- if (kind === "job" && !within(q.hosted_jobs, u.hosted_jobs)) return { ok: false, wait: true, message: `waiting for one of this account's ${q.hosted_jobs} hosted jobs to finish` };
171- if (kind === "sandbox" && !within(q.sandboxes, u.sandboxes)) return { ok: false, wait: false, message: `this account already has ${q.sandboxes} agent sandboxes open; close one or wait for one to finish` };
192+ let running: number | null = null;
193+ if (!q.exempt) {
194+ // Not usageOf: a start doesn't need the repository count or the storage total (up to 25 repositories asked).
195+ const u = await computeUsage(env, accountId);
196+ if (!within(q.hosted_minutes, u.hosted_minutes))
197+ return { ok: false, wait: false, message: `this account has used its ${q.hosted_minutes} hosted minutes for this month (Actions jobs on hosted runners and agent sandboxes); they renew on the 1st (UTC). Self-hosted runners still work` };
198+ if (kind === "job" && !within(q.hosted_jobs, u.hosted_jobs)) return { ok: false, wait: true, message: `waiting for one of this account's ${q.hosted_jobs} hosted jobs to finish` };
199+ if (kind === "sandbox" && !within(q.sandboxes, u.sandboxes)) return { ok: false, wait: false, message: `this account already has ${q.sandboxes} agent sandboxes open; close one or wait for one to finish` };
200+ running = u.hosted_jobs;
201+ }
202+ if (kind === "job") {
203+ // The fleet's limits hold for every account, exempt ones too.
204+ const fleet = fleetLimits(env);
205+ if ((running ?? (await runningJobs(env, accountId))) >= fleet.share) return { ok: false, wait: true, message: `waiting for one of this account's hosted jobs to finish: one account runs at most ${fleet.share} at once, half of this Diffinery's hosted runners` };
206+ if ((await runningJobs(env)) >= fleet.total) return { ok: false, wait: true, message: `waiting for a hosted runner: all ${fleet.total} of this Diffinery's hosted runners are busy` };
207+ }
172208 return { ok: true };
173209 }
174210
modifiedapps/diffinery/src/repo/actions.ts+9−10View file
@@ -24,7 +24,7 @@
2424 import { HttpError, newId, now, randomToken, sha256 } from "../lib/util.js";
2525 import { localOnly } from "../auth/dev.js";
2626 import { egressFor, type Egress } from "../lib/egress.js";
27-import { endCompute, mayStart, startCompute } from "../lib/quotas.js";
27+import { endCompute, fleetLimits, mayStart, startCompute } from "../lib/quotas.js";
2828 import { blockedBy, killSwitches, type Work } from "../lib/killswitch.js";
2929 import type { Env } from "../env.js";
3030 import type { RepoDO } from "./RepoDO.js";
@@ -68,8 +68,6 @@
6868 /** Jobs in one run, counting each matrix leg and each job of a called workflow: keeps a run's rows and its RunWorkflow steps bounded. */
6969 export const MAX_RUN_JOBS = 1000;
7070 export const MAX_WORKFLOW_BYTES = 512 * 1024;
71-/** Hosted jobs one repository runs at once (GitHub's free plan allows 20 per account); more wait their turn. */
72-export const MAX_HOSTED_JOBS = 20;
7371 /**
7472 * A hosted job whose container never got a machine (the platform had none free, usually when many
7573 * jobs start at once or right after a deploy) goes back in line and starts again later, this many
@@ -1233,6 +1231,13 @@
12331231 this.repo.upsertCheck({ sha: run.head_sha ?? run.sha, name: this.checkName(run, job), status: "queued", runId: run.id, jobId, detailsUrl: `${this.env.PUBLIC_URL}/${m.owner}/${m.name}/actions/runs/${run.id}/job/${jobId}` });
12341232 return null;
12351233 }
1234+ // Already at an account's share of the fleet in this repository alone (lib/quotas.ts fleetLimits):
1235+ // the job waits for one of the repository's own to finish. Checked here first, it needs no D1 read.
1236+ const busy = this.sql.exec<{ n: number }>("SELECT count(*) AS n FROM jobs WHERE labels IS NULL AND container IS NOT NULL AND status IN ('waiting', 'in_progress') AND id != ?", jobId).one().n;
1237+ if (busy >= fleetLimits(this.env).share) {
1238+ this.holdForSlot(job, 1);
1239+ return null;
1240+ }
12361241 // The account that owns the repository pays (lib/quotas.ts): past its monthly minutes the job
12371242 // fails; past its hosted jobs at once it waits (fleet_held = 2, retried by the alarm) for one to end.
12381243 const quota = await mayStart(this.env, m.ownerId, "job").catch((e: unknown) => {
@@ -1244,12 +1249,6 @@
12441249 else await this.completeJob(jobId, { conclusion: "failure", outputs: {}, failure: { message: `The job was not started: ${quota.message}.` } });
12451250 return null;
12461251 }
1247- // One repository can't take the whole fleet: past its share, the job waits for one of its own to finish.
1248- const busy = this.sql.exec<{ n: number }>("SELECT count(*) AS n FROM jobs WHERE labels IS NULL AND container IS NOT NULL AND status IN ('waiting', 'in_progress') AND id != ?", jobId).one().n;
1249- if (busy >= MAX_HOSTED_JOBS) {
1250- this.holdForSlot(job, 1);
1251- return null;
1252- }
12531252 this.sql.exec("UPDATE jobs SET fleet_held = 0 WHERE id = ?", jobId);
12541253 const token = `djt_${m.repoId}_${jobId}_${randomToken(24)}`;
12551254 const { size, note } = this.runnerSize(job, run);
@@ -1637,7 +1636,7 @@
16371636 const j = this.jobRow(jobId);
16381637 if (j.status === "completed") return;
16391638 if (j.status === "queued" && j.fleet_held === 2) {
1640- await this.completeJob(jobId, { conclusion: "failure", outputs: {}, failure: { message: "The job waited 6 hours for one of this account's hosted jobs to finish, and was not started." } }, "timed_out");
1639+ await this.completeJob(jobId, { conclusion: "failure", outputs: {}, failure: { message: "The job waited 6 hours for a hosted runner (one of this account's hosted jobs to finish, or a free runner), and was not started." } }, "timed_out");
16411640 return;
16421641 }
16431642 await this.stopContainer(j);
modifiedapps/diffinery/src/repo/schema.ts+1−1View file
@@ -196,7 +196,7 @@
196196 ["jobs", "concurrency_cancel", "INTEGER NOT NULL DEFAULT 0"],
197197 // Characters of log text stored so far (repo/actions.ts MAX_JOB_LOG).
198198 ["jobs", "log_chars", "INTEGER NOT NULL DEFAULT 0"],
199- // 1 while a queued job waits for one of the repository's hosted runner slots (repo/actions.ts MAX_HOSTED_JOBS).
199+ // 1 while a queued job waits for one of the repository's hosted runner slots (an account's share, lib/quotas.ts fleetLimits).
200200 ["jobs", "fleet_held", "INTEGER NOT NULL DEFAULT 0"],
201201 // The registered self-hosted runner that claimed the job (null for hosted jobs and personal-token claims).
202202 ["jobs", "runner_id", "INTEGER"],
modifiedapps/diffinery/test/unit/actions-runtime.test.ts+13−9View file
@@ -1,7 +1,8 @@
11 import { describe, expect, it } from "vitest";
22 import { BodyTooLarge, STEP_OPS, batchParts, blockKey, chunkKey, compose, finalKey, listStaged, parseBlockList, parseRange, planCompose, putStream, runComposeStep, stagingPrefix, type BlobBucket, type ComposeLimits, type ComposeStep } from "../../src/lib/blob-store.js";
33 import { DatabaseSync } from "node:sqlite";
4-import { Actions, MAX_ACTIVE_RUNS, MAX_HOSTED_JOBS, MAX_PLACEMENT_TRIES, MAX_RUNNER_RESTARTS, parseRunnerInfo, placementDelay, STALE_IMAGE_TRIES } from "../../src/repo/actions.js";
4+import { Actions, MAX_ACTIVE_RUNS, MAX_PLACEMENT_TRIES, MAX_RUNNER_RESTARTS, parseRunnerInfo, placementDelay, STALE_IMAGE_TRIES } from "../../src/repo/actions.js";
5+import { fleetLimits } from "../../src/lib/quotas.js";
56 import { testD1 } from "./stubs/d1.js";
67 import { DEFAULT_ARTIFACT_QUOTA, Runtime, artifactFileName, artifactObjectKeys } from "../../src/repo/runtime.js";
78 import { REPO_SCHEMA, migrateRepoSchema } from "../../src/repo/schema.js";
@@ -15,6 +16,9 @@
1516 import type { Caller, RepoMeta } from "../../src/repo/types.js";
1617 import { numericId } from "../../src/http/gh.js";
1718
19+/** Hosted jobs one account (and so one repository) runs at once by default. */
20+const SHARE = fleetLimits({} as Env).share;
21+
1822 const MASTER = { MASTER_KEY: b64(new Uint8Array(32).fill(7)) };
1923 const OTHER = { MASTER_KEY: b64(new Uint8Array(32).fill(9)) };
2024 const soon = () => Math.floor(Date.now() / 1000) + 600;
@@ -667,7 +671,7 @@
667671 });
668672
669673 describe("hosted runner slots", () => {
670- it("a repository runs at most MAX_HOSTED_JOBS hosted jobs at once; the rest wait their turn", async () => {
674+ it("a repository runs at most SHARE hosted jobs at once; the rest wait their turn", async () => {
671675 const sql = sqlStorage();
672676 sql.exec(REPO_SCHEMA);
673677 migrateRepoSchema(sql as unknown as SqlStorage);
@@ -689,17 +693,17 @@
689693 const actions = new Actions(repo, env);
690694 const t = new Date().toISOString();
691695 sql.exec("INSERT INTO runs (id, run_number, workflow_path, workflow_name, event, ref, sha, actor, status, created_at, updated_at, event_payload) VALUES ('run1', 1, 'w.yml', 'w', 'push', 'refs/heads/main', 'abc', '{}', 'in_progress', ?, ?, '{}')", t, t);
692- for (let i = 0; i < MAX_HOSTED_JOBS + 2; i++) sql.exec("INSERT INTO jobs (id, run_id, job_key, name, status) VALUES (?, 'run1', ?, ?, 'queued')", `j${i}`, `k${i}`, `k${i}`);
696+ for (let i = 0; i < SHARE + 2; i++) sql.exec("INSERT INTO jobs (id, run_id, job_key, name, status) VALUES (?, 'run1', ?, ?, 'queued')", `j${i}`, `k${i}`, `k${i}`);
693697 const launch = (id: string) => (actions as unknown as { launch(id: string): Promise<void> }).launch(id);
694- for (let i = 0; i < MAX_HOSTED_JOBS + 2; i++) await launch(`j${i}`);
695- expect(started).toHaveLength(MAX_HOSTED_JOBS);
698+ for (let i = 0; i < SHARE + 2; i++) await launch(`j${i}`);
699+ expect(started).toHaveLength(SHARE);
696700 const held = () => sql.exec("SELECT id FROM jobs WHERE fleet_held = 1 AND status = 'queued' ORDER BY rowid").toArray().map((r) => r.id);
697- expect(held()).toEqual([`j${MAX_HOSTED_JOBS}`, `j${MAX_HOSTED_JOBS + 1}`]);
701+ expect(held()).toEqual([`j${SHARE}`, `j${SHARE + 1}`]);
698702 // A finished job hands its slot to the oldest one waiting.
699703 await (actions as unknown as { completeJob(id: string, c: unknown): Promise<void> }).completeJob("j0", { conclusion: "success", outputs: {} });
700- expect(started).toHaveLength(MAX_HOSTED_JOBS + 1);
701- expect(started.at(-1)).toBe(`job-j${MAX_HOSTED_JOBS}`);
702- expect(held()).toEqual([`j${MAX_HOSTED_JOBS + 1}`]);
704+ expect(started).toHaveLength(SHARE + 1);
705+ expect(started.at(-1)).toBe(`job-j${SHARE}`);
706+ expect(held()).toEqual([`j${SHARE + 1}`]);
703707 });
704708
705709 it("a job whose container gets no machine waits and starts again, and fails only after MAX_PLACEMENT_TRIES", async () => {
modifiedapps/diffinery/test/unit/launch.test.ts+10−6View file
@@ -1,13 +1,17 @@
11 import { describe, expect, it } from "vitest";
22 import { DatabaseSync } from "node:sqlite";
33 import { parseWorkflow } from "@diffinery/workflow-yaml";
4-import { Actions, MAX_HOSTED_JOBS } from "../../src/repo/actions.js";
4+import { Actions } from "../../src/repo/actions.js";
5+import { fleetLimits } from "../../src/lib/quotas.js";
56 import { REPO_SCHEMA, migrateRepoSchema } from "../../src/repo/schema.js";
67 import { testD1 } from "./stubs/d1.js";
78 import type { RepoDO } from "../../src/repo/RepoDO.js";
89 import type { Env } from "../../src/env.js";
910 import type { Caller } from "../../src/repo/types.js";
1011
12+/** Hosted jobs one account (and so one repository) runs at once by default. */
13+const SHARE = fleetLimits({} as Env).share;
14+
1115 /** The Durable Object's SqlStorage over node:sqlite. */
1216 function sqlStorage() {
1317 const db = new DatabaseSync(":memory:");
@@ -108,17 +112,17 @@
108112 it("starts several containers at once, up to the repository's share, and holds the rest", async () => {
109113 const f = fixture({ startMs: 20 });
110114 f.DB.sqlite.prepare("INSERT INTO account_quotas (account_id, hosted_jobs, updated_at, updated_by) VALUES ('acme', -1, ?, 'test')").run(new Date().toISOString());
111- const legs = MAX_HOSTED_JOBS + 6;
115+ const legs = SHARE + 6;
112116 const t = Date.now();
113117 const { jobIds } = await f.start(legs);
114118 expect(jobIds).toHaveLength(legs);
115- expect(f.started).toHaveLength(MAX_HOSTED_JOBS);
116- // Not one after another: 20 starts of 20 ms each took far less than 400 ms.
119+ expect(f.started).toHaveLength(SHARE);
120+ // Not one after another: SHARE starts of 20 ms each took far less than SHARE * 20 ms.
117121 expect(f.most()).toBeGreaterThan(1);
118122 expect(f.most()).toBeLessThanOrEqual(8);
119- expect(Date.now() - t).toBeLessThan(MAX_HOSTED_JOBS * 20);
123+ expect(Date.now() - t).toBeLessThan(SHARE * 20);
120124 const rows = f.jobs();
121- expect(rows.filter((j) => j.status === "waiting")).toHaveLength(MAX_HOSTED_JOBS);
125+ expect(rows.filter((j) => j.status === "waiting")).toHaveLength(SHARE);
122126 expect(rows.filter((j) => j.status === "queued" && j.fleet_held === 1)).toHaveLength(6);
123127 });
124128
modifiedapps/diffinery/test/unit/quotas.test.ts+28−1View file
@@ -1,5 +1,5 @@
11 import { describe, expect, it } from "vitest";
2-import { checkRepositoryQuota, checkStorageQuota, DEFAULT_QUOTAS, endCompute, mayStart, monthStart, quotasFor, setQuotas, startCompute, storageRefusal, storedBytes, usageOf } from "../../src/lib/quotas.js";
2+import { checkRepositoryQuota, checkStorageQuota, DEFAULT_QUOTAS, endCompute, fleetLimits, HOSTED_JOBS_LIMIT, mayStart, monthStart, quotasFor, setQuotas, startCompute, storageRefusal, storedBytes, usageOf } from "../../src/lib/quotas.js";
33 import type { Env } from "../../src/env.js";
44 import { testD1 } from "./stubs/d1.js";
55
@@ -75,6 +75,33 @@
7575 expect(await mayStart(env, "ops", "job")).toEqual({ ok: true });
7676 });
7777
78+ it("holds every account, exempt ones too, to half the fleet, and the fleet to its limit", async () => {
79+ expect(fleetLimits({} as Env)).toEqual({ total: HOSTED_JOBS_LIMIT, share: HOSTED_JOBS_LIMIT / 2 });
80+ expect(fleetLimits({ HOSTED_JOBS_LIMIT: "9" } as Env)).toEqual({ total: 9, share: 4 });
81+ for (const junk of ["", "1", "-1", "lots"]) expect(fleetLimits({ HOSTED_JOBS_LIMIT: junk } as Env).total).toBe(HOSTED_JOBS_LIMIT);
82+ const { env, usage } = setup({ HOSTED_JOBS_LIMIT: "6" });
83+ const recent = new Date(Date.now() - 60_000).toISOString();
84+ await setQuotas(env, "acme", { hosted_jobs: -1 }, "root");
85+ // ops is run by a site administrator: no quota of its own, but still half the fleet (3 of 6).
86+ for (const id of ["o1", "o2"]) usage(id, "ops", "job", recent, null);
87+ expect(await mayStart(env, "ops", "job")).toEqual({ ok: true });
88+ usage("o3", "ops", "job", recent, null);
89+ expect(await mayStart(env, "ops", "job")).toMatchObject({ ok: false, wait: true, message: expect.stringMatching(/one account runs at most 3 at once, half of this Diffinery's hosted runners/) });
90+ // A sandbox isn't a job: the fleet's limits don't apply to it.
91+ expect(await mayStart(env, "ops", "sandbox")).toEqual({ ok: true });
92+ // acme, with no limit of its own, gets the other half; then the fleet is full for anyone.
93+ for (const id of ["a1", "a2"]) usage(id, "acme", "job", recent, null);
94+ expect(await mayStart(env, "acme", "job")).toEqual({ ok: true });
95+ usage("a3", "acme", "job", recent, null);
96+ expect(await mayStart(env, "ann", "job")).toMatchObject({ ok: false, wait: true, message: expect.stringMatching(/all 6 of this Diffinery's hosted runners are busy/) });
97+ // A job that ends frees its place, and a row left open past a job's longest run no longer counts.
98+ await endCompute(env, "o3");
99+ expect(await mayStart(env, "ann", "job")).toEqual({ ok: true });
100+ expect(await mayStart(env, "ops", "job")).toEqual({ ok: true });
101+ usage("stale", "ann", "job", new Date(Date.now() - 8 * 3600_000).toISOString(), null);
102+ expect(await mayStart(env, "ann", "job")).toEqual({ ok: true });
103+ });
104+
78105 it("caps the repositories an account owns", async () => {
79106 const { env, DB } = setup();
80107 await setQuotas(env, "ann", { repositories: 1 }, "root");