Bounded matrices, job logs, workflow files and sandboxes; job tokens keep to their repository's packages (#80)
* Matrices are bounded before they're expanded, and job logs and workflow files have size limits A matrix of 20 values in each of 6 vectors made 64 million combinations before the 256-job check, holding a repository's Durable Object for minutes. A job's log grew without limit and was read into memory whole when the job completed. Every push parsed every workflow file, of any size. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016yhgAu6oKfqes812Yd4qjo * A workflow's token can't use another repository's private packages, and a repository's name is its own in the registry A job token with packages: read pulled every private package of its owner, including images linked to repositories the job's repository has no access to. Anyone who could push to an owner could also create a package named after one of its repositories before that repository's workflow did. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016yhgAu6oKfqes812Yd4qjo * One account keeps at most 5 sandboxes open in a repository Sandbox containers are shared by the whole instance (50 at once), and one account could open them until none were left for anyone else. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016yhgAu6oKfqes812Yd4qjo --------- Co-authored-by: Claude <noreply@anthropic.com>
Canberk Demirci authored · GitHub committed signed1 parent d59844fcommit 526abbd9431a3fe2f87589a4f085894d0593dfb4Browse files
11 files changed+178−15
Files (11)
- apps/diffinery/docs/actions/workflow-syntax.md+2−1
- apps/diffinery/docs/packages/container-registry.md+5−5
- apps/diffinery/docs/reference/limits.md+6−0
- apps/diffinery/src/registry/access.ts+14−5
- apps/diffinery/src/repo/actions.ts+37−4
- apps/diffinery/src/repo/objects.ts+5−0
- apps/diffinery/src/repo/schema.ts+2−0
- apps/diffinery/test/e2e/security.test.mjs+67−0
- apps/diffinery/test/e2e/tasks.test.mjs+10−0
- packages/workflow-yaml/src/matrix.ts+12−0
- packages/workflow-yaml/test/matrix.test.ts+18−0
modifiedapps/diffinery/docs/actions/workflow-syntax.md+2−1View file
| @@ -467,9 +467,10 @@ | ||
| 467 | 467 | - A matrix with only `include` produces exactly the `include` entries. |
| 468 | 468 | - The whole matrix, any list, `include` or `exclude` can be a single `${{ }}` expression, for example `matrix: ${{ fromJSON(needs.plan.outputs.matrix) }}`. Such expressions are evaluated when the job starts, with the `github`, `needs`, `inputs`, `vars` and `env` contexts. |
| 469 | 469 | - A matrix may produce at most 256 jobs. A larger one fails the job with `A matrix may produce at most 256 jobs.` |
| 470 | +- Before `exclude` is applied, the vectors may have at most 4,096 combinations, and `include` and `exclude` at most 256 entries each. A larger matrix is refused before it is expanded (at parse time, or at run time for one built with `fromJSON`). | |
| 470 | 471 | - The `matrix` and `strategy` contexts give each job its values (`strategy.job-index`, `strategy.job-total`, `strategy.fail-fast`, `strategy.max-parallel`). |
| 471 | 472 | |
| 472 | -Parse and runtime errors: `matrix produces no job combinations`, `matrix.<key> must be a list (got <type>)`, `matrix.include must be a list of mappings`, `matrix.exclude[<n>] must be a mapping`, `matrix must be a mapping or a single ${{ }} expression`, and at runtime `Error expanding matrix: <message>`. | |
| 473 | +Parse and runtime errors: `matrix produces no job combinations`, `matrix.<key> must be a list (got <type>)`, `matrix.include must be a list of mappings`, `matrix.exclude[<n>] must be a mapping`, `matrix must be a mapping or a single ${{ }} expression`, `the matrix has <n> combinations before exclusions; at most 4096 are allowed (and a matrix may produce at most 256 jobs)`, `matrix.include has <n> entries; at most 256 are allowed`, and at runtime `Error expanding matrix: <message>`. | |
| 473 | 474 | |
| 474 | 475 | #### fail-fast |
| 475 | 476 |
modifiedapps/diffinery/docs/packages/container-registry.md+5−5View file
| @@ -131,7 +131,7 @@ | ||
| 131 | 131 | A new package is linked when it's created in one of these ways: |
| 132 | 132 | |
| 133 | 133 | - **By a workflow.** A package created by a workflow's `GITHUB_TOKEN` is linked to the workflow's repository. |
| 134 | -- **By its name.** A person or agent who pushes `diffinery.com/<owner>/<repo>` or `diffinery.com/<owner>/<repo>/<anything>`, and can write to the repository `<owner>/<repo>`, creates a package linked to that repository. For example, pushing `diffinery.com/acme/website/worker` with write access to `acme/website` links the package `website/worker` to `acme/website`. | |
| 134 | +- **By its name.** A person or agent who pushes `diffinery.com/<owner>/<repo>` or `diffinery.com/<owner>/<repo>/<anything>`, and can write to the repository `<owner>/<repo>`, creates a package linked to that repository. For example, pushing `diffinery.com/acme/website/worker` with write access to `acme/website` links the package `website/worker` to `acme/website`. Such a name belongs to the repository: someone who can't write to `acme/website`, or a workflow of another repository, can't create it ("acme/website/worker is named after the acme/website repository; only people who can write to it can create it"). | |
| 135 | 135 | - **By a label.** When an image has the label or annotation `org.opencontainers.image.source` set to a repository URL of the same owner, on `https://diffinery.com` or `https://github.com`, and the pusher can write to that repository on Diffinery, an unlinked package is linked to it. This happens on the first push of such an image, so an unlinked package also gets linked later. |
| 136 | 136 | |
| 137 | 137 | ```dockerfile |
| @@ -227,8 +227,8 @@ | ||
| 227 | 227 | | --- | --- | --- | |
| 228 | 228 | | Linked to the workflow's repository | Pull | Admin: push and delete versions | |
| 229 | 229 | | Same owner as the workflow's repository, not linked to any repository | Pull | Pull | |
| 230 | -| Same owner, linked to another repository | Pull | Pull | | |
| 231 | -| A new name under the same owner | Can't create | Push: creates the package, linked to the workflow's repository | | |
| 230 | +| Same owner, linked to another repository | None (Pull if public) | None (Pull if public) | | |
| 231 | +| A new name under the same owner | Can't create | Push: creates the package, linked to the workflow's repository, unless the name's first part is another repository's name | | |
| 232 | 232 | | Another owner's public package | Pull | Pull | |
| 233 | 233 | | Another owner's private package, or a new name under another owner | None | None | |
| 234 | 234 | |
| @@ -318,7 +318,7 @@ | ||
| 318 | 318 | - run: curl -fsS http://api:8080/health |
| 319 | 319 | ``` |
| 320 | 320 | |
| 321 | -With `packages: read`, a workflow can pull every package of its own owner and every public package. A later job of the same run (or another workflow) can also pull an image that an earlier job pushed in a `run:` step: | |
| 321 | +With `packages: read`, a workflow can pull its own repository's packages, its owner's packages that aren't linked to a repository, and every public package. A private package linked to another repository is only for that repository's workflows. A later job of the same run (or another workflow) can also pull an image that an earlier job pushed in a `run:` step: | |
| 322 | 322 | |
| 323 | 323 | ```yaml |
| 324 | 324 | - run: | |
| @@ -558,7 +558,7 @@ | ||
| 558 | 558 | ## Differences from ghcr.io |
| 559 | 559 | |
| 560 | 560 | - Visibility is private or public. A public package is pullable by anyone; a private one only by the people and workflows described in [Permissions](#permissions). |
| 561 | -- A workflow's token with `packages: read` can pull every package of its owner. | |
| 561 | +- A workflow's token with `packages: read` can pull its owner's packages that aren't linked to any repository; ghcr.io asks for each package to grant the repository access first. | |
| 562 | 562 | - The link to a repository can be changed after creation, on the package page or with `PATCH`. |
| 563 | 563 | - Package settings, visibility and deletion are on the package page; there is no separate access list per package. |
| 564 | 564 | - A single request is limited to 100 MB outside hosted runners. |
modifiedapps/diffinery/docs/reference/limits.md+6−0View file
| @@ -167,6 +167,10 @@ | ||
| 167 | 167 | | Step timeout | No maximum beyond the job's | `steps[*].timeout-minutes`. | |
| 168 | 168 | | Waiting for a concurrency group | 6 hours | A run whose `concurrency` group is busy (without `cancel-in-progress`) checks once a minute for up to 6 hours. | |
| 169 | 169 | | Jobs in one matrix | 256 | More combinations fail the job with "A matrix may produce at most 256 jobs." | |
| 170 | +| Matrix combinations before `exclude` | 4,096 | The product of the matrix's vectors. A larger matrix is refused without being expanded: "the matrix has <n> combinations before exclusions; at most 4096 are allowed (and a matrix may produce at most 256 jobs)". | | |
| 171 | +| Matrix `include` and `exclude` entries | 256 each | "matrix.include has <n> entries; at most 256 are allowed". | | |
| 172 | +| Workflow files read per commit | 256 | Of `.github/workflows` and `.diffinery/workflows` together, in name order; later files are ignored. | | |
| 173 | +| Workflow file size | 512 KiB | A larger file is listed as invalid ("the workflow file is larger than 512 KiB") and never runs. | | |
| 170 | 174 | | Reusable workflow nesting | 4 levels | "reusable workflows can be nested at most 4 levels deep" | |
| 171 | 175 | | Reusable workflows per run | 20 calls | "a run can call at most 20 reusable workflows" | |
| 172 | 176 | | `workflow_run` chains | 3 levels | A run started by `workflow_run` that is itself three links down a chain triggers nothing (A starts B starts C starts D; D's runs start nothing). | |
| @@ -178,6 +182,7 @@ | ||
| 178 | 182 | | Jobs during a deploy | Not interrupted | A deploy doesn't replace the container of a running hosted job: it finishes on the version it started with. If the platform replaces a job's container anyway, the job keeps going for up to 13 minutes, then is cancelled. | |
| 179 | 183 | | Service container health check | 15 minutes | How long a job waits for a `services:` container's health check to pass. | |
| 180 | 184 | | Log line length | 16,384 characters | Longer lines are cut and end with `…`. | |
| 185 | +| Log size per job | 16 MiB of text | The log ends with "The log is cut off here: a job's log is limited to 16 MiB of text." and later lines are dropped; the job keeps running. | | |
| 181 | 186 | | Log shown in the web UI | 20,000 lines | The job page shows the first 20,000 lines and links to the full log. | |
| 182 | 187 | | Job log retention | 90 days | A finished job's log is stored for 90 days, then deleted. The run and its jobs stay listed. | |
| 183 | 188 | | OIDC token lifetime | 10 minutes | ID tokens from `permissions: id-token: write`. | |
| @@ -209,6 +214,7 @@ | ||
| 209 | 214 | | --- | --- | --- | |
| 210 | 215 | | Sandbox size | Cloudflare Containers `standard-1` | Each sandbox runs in its own container of this instance type. | |
| 211 | 216 | | Sandboxes at once | 50 | `max_instances` of the sandbox container class, across the whole instance. | |
| 217 | +| Open sandboxes per account and repository | 5 | A sixth is refused with `429 you have 5 open sandboxes in this repository, the most allowed; close one first`. A sandbox counts until it is closed or its credentials expire after 24 hours. Agent task sandboxes don't count. | | |
| 212 | 218 | | Start-up wait | Up to 3 minutes | Creating a sandbox waits up to 3 minutes for its container (longer than usual right after a deploy, while the host pulls the image). If it doesn't start, creation fails with `503 the sandbox container did not start; try again`. | |
| 213 | 219 | | Idle time | 30 minutes | A sandbox that receives no requests for 30 minutes is stopped. | |
| 214 | 220 | | Command timeout | 10 minutes by default | `exec` in the REST API (`timeoutMs`) and `sandbox_exec` in MCP (`timeout_seconds`) can set another value. A command that times out returns exit code `124` and `timedOut: true`. | |
modifiedapps/diffinery/src/registry/access.ts+14−5View file
| @@ -65,12 +65,18 @@ | ||
| 65 | 65 | let hint: string | undefined; |
| 66 | 66 | if (pkg) { |
| 67 | 67 | const linked = pkg.repo_id === repo.id; |
| 68 | - if (perm !== "none" && (linked || sameOwner || pkg.visibility === "public")) role = "pull"; | |
| 68 | + // A package linked to another repository follows that repository's access, which a job token | |
| 69 | + // doesn't have: like ghcr.io, only that repository's workflows use it (unless it is public). | |
| 70 | + if (perm !== "none" && (linked || (sameOwner && !pkg.repo_id) || pkg.visibility === "public")) role = "pull"; | |
| 69 | 71 | if (perm === "write" && linked) role = "admin"; |
| 70 | - else if (pkg.repo_id && !linked) hint = `${owner.login}/${pkg.name} belongs to the ${owner.login}/${pkg.repo_name ?? "?"} repository, so only its workflows can push to it`; | |
| 72 | + else if (pkg.repo_id && !linked) hint = `${owner.login}/${pkg.name} belongs to the ${owner.login}/${pkg.repo_name ?? "?"} repository, so only its workflows can use it`; | |
| 71 | 73 | else if (!pkg.repo_id) hint = `${owner.login}/${pkg.name} isn't linked to a repository; link it to ${repo.owner_login}/${repo.name} in its settings so this workflow can push to it`; |
| 72 | - } else if (perm === "write" && sameOwner) role = "push"; | |
| 73 | - else if (!sameOwner) hint = `workflows in ${repo.owner_login}/${repo.name} can only create packages under ${repo.owner_login}`; | |
| 74 | + } else if (perm === "write" && sameOwner) { | |
| 75 | + // A new name that is another repository's (diffinery.com/<owner>/<repo>[/...]) is left for that repository. | |
| 76 | + const named = await dir.repo(owner.login, name.split("/")[0]!); | |
| 77 | + if (named && named.id !== repo.id) hint = `${owner.login}/${name} is named after the ${owner.login}/${named.name} repository, so only its workflows can create it`; | |
| 78 | + else role = "push"; | |
| 79 | + } else if (!sameOwner) hint = `workflows in ${repo.owner_login}/${repo.name} can only create packages under ${repo.owner_login}`; | |
| 74 | 80 | if (perm === "none") hint = "the workflow's GITHUB_TOKEN has no packages permission; add `permissions: packages: write`"; |
| 75 | 81 | else if (perm === "read" && !hint) hint = "the workflow's GITHUB_TOKEN has packages: read; add `permissions: packages: write` to push"; |
| 76 | 82 | return { role, linkRepo: repo, actor, hint }; |
| @@ -108,13 +114,16 @@ | ||
| 108 | 114 | const repo = await dir.repoById(pkg.repo_id); |
| 109 | 115 | if (repo) role = max(role, fromRepoRole(await dir.roleFor(a, repo))); |
| 110 | 116 | } else if (!pkg) { |
| 111 | - // diffinery.com/<owner>/<repo>[/...] is linked to that repository when the pusher can write to it. | |
| 117 | + // diffinery.com/<owner>/<repo>[/...] is linked to that repository when the pusher can write to it, | |
| 118 | + // and is that repository's name: someone who can't write to it can't take the name first. | |
| 112 | 119 | const repo = await dir.repo(owner.login, name.split("/")[0]!); |
| 113 | 120 | if (repo) { |
| 114 | 121 | const r = fromRepoRole(await dir.roleFor(a, repo)); |
| 115 | 122 | if (allows(r, "push")) { |
| 116 | 123 | linkRepo = repo; |
| 117 | 124 | role = max(role, r); |
| 125 | + } else if (!a.site_admin) { | |
| 126 | + return { role: "none", linkRepo: null, actor, hint: `${owner.login}/${name} is named after the ${owner.login}/${repo.name} repository; only people who can write to it can create it` }; | |
| 118 | 127 | } |
| 119 | 128 | } |
| 120 | 129 | } |
modifiedapps/diffinery/src/repo/actions.ts+37−4View file
| @@ -35,8 +35,15 @@ | ||
| 35 | 35 | |
| 36 | 36 | /** How long a sandbox token works after the sandbox starts. */ |
| 37 | 37 | export const SANDBOX_TOKEN_TTL_MS = 24 * 3600_000; |
| 38 | +/** Sandboxes one person or agent may have open in a repository at once: their containers are shared by the whole instance. */ | |
| 39 | +export const MAX_OPEN_SANDBOXES = 5; | |
| 38 | 40 | |
| 39 | 41 | const WORKFLOW_DIRS = [".github/workflows", ".diffinery/workflows"]; |
| 42 | +/** Characters of text a job's log keeps; the rest is dropped after a line saying so. */ | |
| 43 | +export const MAX_JOB_LOG = 16 * 1048576; | |
| 44 | +/** Every push reads and parses a commit's workflow files: at most this many, each at most this large. */ | |
| 45 | +export const MAX_WORKFLOW_FILES = 256; | |
| 46 | +export const MAX_WORKFLOW_BYTES = 512 * 1024; | |
| 40 | 47 | const SUPPORTED_LABEL = /^(ubuntu-.*|linux|diffinery(-.*)?|self-hosted|x64|amd64)$/i; |
| 41 | 48 | |
| 42 | 49 | export type RunRow = { |
| @@ -90,6 +97,7 @@ | ||
| 90 | 97 | /** Job-level `concurrency` group as evaluated for this leg, and its cancel-in-progress (0 or 1). */ |
| 91 | 98 | concurrency_group: string | null; |
| 92 | 99 | concurrency_cancel: number; |
| 100 | + log_chars: number; | |
| 93 | 101 | }; |
| 94 | 102 | |
| 95 | 103 | export type RunInfo = { |
| @@ -191,7 +199,12 @@ | ||
| 191 | 199 | if (!entry || !isTreeMode(entry.mode)) continue; |
| 192 | 200 | for (const f of await readTree(this.repo.store, entry.oid)) { |
| 193 | 201 | if (!/\.ya?ml$/.test(f.name) || isTreeMode(f.mode)) continue; |
| 202 | + if (out.length >= MAX_WORKFLOW_FILES) break; | |
| 194 | 203 | const path = `${dir}/${f.name}`; |
| 204 | + if ((this.repo.store.size(f.oid) ?? 0) > MAX_WORKFLOW_BYTES) { | |
| 205 | + out.push({ path, errors: [`: the workflow file is larger than ${MAX_WORKFLOW_BYTES / 1024} KiB`] }); | |
| 206 | + continue; | |
| 207 | + } | |
| 195 | 208 | const o = await this.repo.store.read(f.oid); |
| 196 | 209 | if (!o) continue; |
| 197 | 210 | const { workflow, errors } = parseWorkflow(fromUtf8(o.content), path); |
| @@ -1267,15 +1280,28 @@ | ||
| 1267 | 1280 | } |
| 1268 | 1281 | |
| 1269 | 1282 | async runnerLogs(jobId: string, token: string, lines: LogLine[]): Promise<void> { |
| 1270 | - await this.authJob(jobId, token); | |
| 1283 | + const job = await this.authJob(jobId, token); | |
| 1284 | + let chars = job.log_chars ?? 0; | |
| 1285 | + if (chars > MAX_JOB_LOG) return; | |
| 1271 | 1286 | let seq = this.sql.exec<{ n: number | null }>("SELECT max(seq) AS n FROM logs WHERE job_id = ?", jobId).one().n ?? -1; |
| 1272 | 1287 | const out: Array<LogLine & { seq: number }> = []; |
| 1273 | 1288 | for (const l of lines.slice(0, 5000)) { |
| 1274 | 1289 | seq++; |
| 1275 | - const text = l.text.length > 16384 ? `${l.text.slice(0, 16384)}…` : l.text; | |
| 1276 | - this.sql.exec("INSERT INTO logs (job_id, seq, t, step, text) VALUES (?, ?, ?, ?, ?)", jobId, seq, l.t, l.step, text); | |
| 1277 | - out.push({ ...l, text, seq }); | |
| 1290 | + const t = String(l.t ?? "").slice(0, 40); | |
| 1291 | + const step = Number.isInteger(l.step) ? l.step : 0; | |
| 1292 | + let text = String(l.text ?? ""); | |
| 1293 | + if (text.length > 16384) text = `${text.slice(0, 16384)}…`; | |
| 1294 | + // Past the cap, one last line says so and the rest is dropped: the whole log is read into | |
| 1295 | + // memory when the job completes, to archive it. | |
| 1296 | + if (chars + text.length > MAX_JOB_LOG) { | |
| 1297 | + text = `The log is cut off here: a job's log is limited to ${MAX_JOB_LOG / 1048576} MiB of text.`; | |
| 1298 | + chars = MAX_JOB_LOG + 1; | |
| 1299 | + } else chars += text.length; | |
| 1300 | + this.sql.exec("INSERT INTO logs (job_id, seq, t, step, text) VALUES (?, ?, ?, ?, ?)", jobId, seq, t, step, text); | |
| 1301 | + out.push({ t, step, text, seq }); | |
| 1302 | + if (chars > MAX_JOB_LOG) break; | |
| 1278 | 1303 | } |
| 1304 | + this.sql.exec("UPDATE jobs SET log_chars = ? WHERE id = ?", chars, jobId); | |
| 1279 | 1305 | this.repo.broadcast(`job:${jobId}`, { action: "logs", lines: out }); |
| 1280 | 1306 | } |
| 1281 | 1307 | |
| @@ -1545,6 +1571,13 @@ | ||
| 1545 | 1571 | if (!atLeast(caller.role, "write")) throw new HttpError(403, "write access is required to create a sandbox"); |
| 1546 | 1572 | // A review's sandbox acts with read access only: its git credential can clone but never push. |
| 1547 | 1573 | if (task?.readOnly) caller = { ...caller, role: "read" }; |
| 1574 | + if (!task) { | |
| 1575 | + const since = new Date(Date.now() - SANDBOX_TOKEN_TTL_MS).toISOString(); | |
| 1576 | + const open = this.sql | |
| 1577 | + .exec<{ n: number }>("SELECT count(*) AS n FROM sandboxes WHERE kind = 'api' AND status = 'running' AND created_at > ? AND json_extract(created_by, '$.id') = ?", since, caller.id) | |
| 1578 | + .one().n; | |
| 1579 | + if (open >= MAX_OPEN_SANDBOXES) throw new HttpError(429, `you have ${open} open sandboxes in this repository, the most allowed; close one first`); | |
| 1580 | + } | |
| 1548 | 1581 | const sha = await this.repo.resolve(ref ?? "HEAD"); |
| 1549 | 1582 | if (!sha) throw new HttpError(422, "unknown ref"); |
| 1550 | 1583 | const id = newId(); |
modifiedapps/diffinery/src/repo/objects.ts+5−0View file
| @@ -35,6 +35,11 @@ | ||
| 35 | 35 | } |
| 36 | 36 | } |
| 37 | 37 | |
| 38 | + /** An object's inflated size without reading it, or null when it isn't stored. */ | |
| 39 | + size(oid: string): number | null { | |
| 40 | + return this.sql.exec<{ size: number }>("SELECT size FROM objects WHERE oid = ?", oid).toArray()[0]?.size ?? null; | |
| 41 | + } | |
| 42 | + | |
| 38 | 43 | has(oid: string): boolean { |
| 39 | 44 | return this.sql.exec("SELECT 1 FROM objects WHERE oid = ?", oid).toArray().length > 0; |
| 40 | 45 | } |
modifiedapps/diffinery/src/repo/schema.ts+2−0View file
| @@ -175,6 +175,8 @@ | ||
| 175 | 175 | ["jobs", "continue_on_error", "INTEGER NOT NULL DEFAULT 0"], |
| 176 | 176 | ["jobs", "concurrency_group", "TEXT"], |
| 177 | 177 | ["jobs", "concurrency_cancel", "INTEGER NOT NULL DEFAULT 0"], |
| 178 | + // Characters of log text stored so far (repo/actions.ts MAX_JOB_LOG). | |
| 179 | + ["jobs", "log_chars", "INTEGER NOT NULL DEFAULT 0"], | |
| 178 | 180 | // The GitHub import that wrote this row (null for everything else); a retry may only overwrite its own rows. |
| 179 | 181 | ["issues", "imported", "TEXT"], |
| 180 | 182 | ["pulls", "imported", "TEXT"], |
modifiedapps/diffinery/test/e2e/security.test.mjs+67−0View file
| @@ -1743,3 +1743,70 @@ | ||
| 1743 | 1743 | assert.match(repo.text, /Invitation sent to sam@example\.com/); |
| 1744 | 1744 | await status(404, "GET", `/repos/${OWNER}/by-mail-repo`, undefined, T.sam); |
| 1745 | 1745 | }); |
| 1746 | + | |
| 1747 | +test("security: a job's log is capped, and oversized workflow files aren't parsed", async () => { | |
| 1748 | + const R = `/repos/${OWNER}/big-logs`; | |
| 1749 | + await repoWithReadme("big-logs"); | |
| 1750 | + const wf = "name: ci\non: push\njobs:\n build:\n runs-on: [self-hosted, linux]\n steps:\n - run: echo hi\n"; | |
| 1751 | + await ok("PUT", `${R}/contents/.github/workflows/ci.yml`, { message: "ci", content: b64(wf) }); | |
| 1752 | + let claim; | |
| 1753 | + await until("the job claimed", async () => { | |
| 1754 | + const r = await api("POST", `${R}/actions/runners/claim`, { name: "log-runner", labels: ["linux"] }); | |
| 1755 | + if (r.status === 200) claim = r.json; | |
| 1756 | + return !!claim; | |
| 1757 | + }); | |
| 1758 | + const headers = { authorization: `Bearer ${claim.token}`, "content-type": "application/json" }; | |
| 1759 | + const text = "x".repeat(16384); | |
| 1760 | + const post = async (n) => { | |
| 1761 | + const r = await fetch(`${base}/runner/jobs/${claim.job_id}/logs`, { method: "POST", headers, body: JSON.stringify({ lines: Array.from({ length: n }, () => ({ t: new Date().toISOString(), step: 0, text })) }) }); | |
| 1762 | + assert.equal(r.status, 200, await r.text()); | |
| 1763 | + }; | |
| 1764 | + // 1,200 lines of 16 KiB: the log keeps 16 MiB of them, then one line saying it was cut off; later lines are dropped. | |
| 1765 | + await post(600); | |
| 1766 | + await post(600); | |
| 1767 | + await post(10); | |
| 1768 | + await fetch(`${base}/runner/jobs/${claim.job_id}/complete`, { method: "POST", headers, body: JSON.stringify({ conclusion: "success", outputs: {} }) }); | |
| 1769 | + const lines = await ok("GET", `${R}/actions/jobs/${claim.job_id}/logs?format=json`); | |
| 1770 | + assert.equal(lines.length, 1025); | |
| 1771 | + assert.match(lines.at(-1).text, /cut off here: a job's log is limited to 16 MiB/); | |
| 1772 | + | |
| 1773 | + // A workflow file over 512 KiB is listed as invalid without being parsed. | |
| 1774 | + await ok("PUT", `${R}/contents/.github/workflows/huge.yml`, { message: "huge", content: b64(`${wf}# ${"y".repeat(600 * 1024)}\n`) }); | |
| 1775 | + const { workflows } = await ok("GET", `${R}/actions/workflows`); | |
| 1776 | + const huge = workflows.find((w) => w.path === ".github/workflows/huge.yml"); | |
| 1777 | + assert.equal(huge.state, "invalid"); | |
| 1778 | + assert.match(huge.errors.join("\n"), /larger than 512 KiB/); | |
| 1779 | +}); | |
| 1780 | + | |
| 1781 | +test("security: a workflow's token can't use another repository's private packages, and a repository's name is its own", async () => { | |
| 1782 | + await repoWithReadme("pkg-home", { private: true }); | |
| 1783 | + await repoWithReadme("pkg-other", { private: true }); | |
| 1784 | + // olive pushes an image named after pkg-home, so it's linked to it and private. | |
| 1785 | + await pushTinyImage(`${OWNER}/pkg-home`, "v1", T.olive); | |
| 1786 | + const wf = "name: ci\non: push\njobs:\n build:\n runs-on: [self-hosted, linux]\n permissions:\n contents: read\n packages: write\n steps:\n - run: echo hi\n"; | |
| 1787 | + await ok("PUT", `/repos/${OWNER}/pkg-other/contents/.github/workflows/ci.yml`, { message: "ci", content: b64(wf) }); | |
| 1788 | + let claim; | |
| 1789 | + await until("the job claimed", async () => { | |
| 1790 | + const r = await api("POST", `/repos/${OWNER}/pkg-other/actions/runners/claim`, { name: "pkg-runner", labels: ["linux"] }); | |
| 1791 | + if (r.status === 200) claim = r.json; | |
| 1792 | + return !!claim; | |
| 1793 | + }); | |
| 1794 | + // The runner reads its job first; that settles the token's permissions. | |
| 1795 | + await fetch(`${base}/runner/jobs/${claim.job_id}`, { headers: { authorization: `Bearer ${claim.token}` } }); | |
| 1796 | + // pkg-other's workflow can't pull pkg-home's image, nor take a name that belongs to pkg-home. | |
| 1797 | + assert.equal((await pullManifest(`${OWNER}/pkg-home`, "v1", claim.token)).status, 404); | |
| 1798 | + const basic = (as) => ({ authorization: `Basic ${Buffer.from(`anyone:${as}`).toString("base64")}`, "content-type": "application/octet-stream" }); | |
| 1799 | + const blob = Buffer.from("squat"); | |
| 1800 | + const digest = `sha256:${createHash("sha256").update(blob).digest("hex")}`; | |
| 1801 | + const squat = await fetch(`${base}/v2/${OWNER}/pkg-home/tools/blobs/uploads/?digest=${digest}`, { method: "POST", body: blob, headers: basic(claim.token) }); | |
| 1802 | + assert.equal(squat.status, 403); | |
| 1803 | + assert.match(await squat.text(), /named after the olive\/pkg-home repository/); | |
| 1804 | + // Its own names still work. | |
| 1805 | + const own = await fetch(`${base}/v2/${OWNER}/pkg-other/tools/blobs/uploads/?digest=${digest}`, { method: "POST", body: blob, headers: basic(claim.token) }); | |
| 1806 | + assert.equal(own.status, 201, await own.text()); | |
| 1807 | + | |
| 1808 | + // A person who can't write to a repository can't take its name either: milo, a guild member, only reads guild/vault. | |
| 1809 | + const taken = await fetch(`${base}/v2/guild/vault/blobs/uploads/?digest=${digest}`, { method: "POST", body: blob, headers: basic(T.milo) }); | |
| 1810 | + assert.equal(taken.status, 403); | |
| 1811 | + assert.match(await taken.text(), /named after the guild\/vault repository/); | |
| 1812 | +}); |
modifiedapps/diffinery/test/e2e/tasks.test.mjs+10−0View file
| @@ -381,6 +381,16 @@ | ||
| 381 | 381 | assert.equal((await api("POST", `${R}/${n}/messages`, { body: "x" }, dst)).status, 403); |
| 382 | 382 | await ok("POST", `/repos/${owner}/tasks-demo/diffinery/sandboxes/${plain.id}/close`, {}, bot); |
| 383 | 383 | |
| 384 | + // One account keeps at most 5 sandboxes open in a repository; closing one makes room. | |
| 385 | + const open = []; | |
| 386 | + for (let i = 0; i < 5; i++) open.push((await ok("POST", `/repos/${owner}/tasks-demo/diffinery/sandboxes`, {}, bot)).id); | |
| 387 | + const sixth = await api("POST", `/repos/${owner}/tasks-demo/diffinery/sandboxes`, {}, bot); | |
| 388 | + assert.equal(sixth.status, 429); | |
| 389 | + assert.match(sixth.json.message, /5 open sandboxes in this repository/); | |
| 390 | + await ok("POST", `/repos/${owner}/tasks-demo/diffinery/sandboxes/${open.pop()}/close`, {}, bot); | |
| 391 | + open.push((await ok("POST", `/repos/${owner}/tasks-demo/diffinery/sandboxes`, {}, bot)).id); | |
| 392 | + for (const id of open) await ok("POST", `/repos/${owner}/tasks-demo/diffinery/sandboxes/${id}/close`, {}, bot); | |
| 393 | + | |
| 384 | 394 | // The agent can stop its own task; the run's token stops working. |
| 385 | 395 | const stopped = await ok("POST", `${R}/${n}/stop`, {}, bot); |
| 386 | 396 | assert.equal(stopped.state, "stopped"); |
modifiedpackages/workflow-yaml/src/matrix.ts+12−0View file
| @@ -71,6 +71,10 @@ | ||
| 71 | 71 | * original matrix values it does not overwrite (values added by earlier |
| 72 | 72 | * includes may be overwritten); if it fits none, it becomes a new combination. |
| 73 | 73 | */ |
| 74 | +/** Combinations before `exclude`, and entries in `include` or `exclude`: these bound the work of expanding a matrix. */ | |
| 75 | +export const MAX_MATRIX_COMBINATIONS = 4096; | |
| 76 | +export const MAX_MATRIX_ENTRIES = 256; | |
| 77 | + | |
| 74 | 78 | export function expandMatrix(matrix: unknown, ctx?: ExprContext): MatrixCombination[] { |
| 75 | 79 | if (matrix === undefined || matrix === null) return [{}]; |
| 76 | 80 | const resolved = resolve(matrix, ctx); |
| @@ -90,6 +94,14 @@ | ||
| 90 | 94 | } |
| 91 | 95 | } |
| 92 | 96 | |
| 97 | + for (const [what, list] of [["include", include], ["exclude", exclude]] as const) { | |
| 98 | + if (list.length > MAX_MATRIX_ENTRIES) throw new MatrixError(`matrix.${what} has ${list.length} entries; at most ${MAX_MATRIX_ENTRIES} are allowed`); | |
| 99 | + } | |
| 100 | + // Count the combinations before building any, so a matrix like 20 values in each of 6 vectors | |
| 101 | + // (64 million) is refused at once instead of filling memory. | |
| 102 | + const product = dims.reduce((n, [, values]) => n * values.length, dims.length ? 1 : 0); | |
| 103 | + if (product > MAX_MATRIX_COMBINATIONS) throw new MatrixError(`the matrix has ${product} combinations before exclusions; at most ${MAX_MATRIX_COMBINATIONS} are allowed (and a matrix may produce at most 256 jobs)`); | |
| 104 | + | |
| 93 | 105 | let combos: MatrixCombination[] = []; |
| 94 | 106 | if (dims.length > 0) { |
| 95 | 107 | combos = [{}]; |
modifiedpackages/workflow-yaml/test/matrix.test.ts+18−0View file
| @@ -147,3 +147,21 @@ | ||
| 147 | 147 | expect(() => expandMatrix([1, 2])).toThrow(/mapping/); |
| 148 | 148 | }); |
| 149 | 149 | }); |
| 150 | + | |
| 151 | +describe("matrix size", () => { | |
| 152 | + it("refuses a matrix with too many combinations before building any", () => { | |
| 153 | + const m: Record<string, number[]> = {}; | |
| 154 | + for (let i = 0; i < 6; i++) m[`k${i}`] = Array.from({ length: 20 }, (_, j) => j); | |
| 155 | + const start = Date.now(); | |
| 156 | + expect(() => expandMatrix(m)).toThrow(/64000000 combinations before exclusions; at most 4096/); | |
| 157 | + expect(Date.now() - start).toBeLessThan(1000); | |
| 158 | + }); | |
| 159 | + it("still allows a large product that stays under the bound", () => { | |
| 160 | + expect(expandMatrix({ a: Array.from({ length: 64 }, (_, i) => i), b: Array.from({ length: 64 }, (_, i) => i) })).toHaveLength(4096); | |
| 161 | + }); | |
| 162 | + it("bounds include and exclude", () => { | |
| 163 | + const many = Array.from({ length: 257 }, (_, i) => ({ a: i })); | |
| 164 | + expect(() => expandMatrix({ a: [1], include: many })).toThrow(/matrix.include has 257 entries/); | |
| 165 | + expect(() => expandMatrix({ a: [1], exclude: many })).toThrow(/matrix.exclude has 257 entries/); | |
| 166 | + }); | |
| 167 | +}); |