Live updates carry no payloads, and push protection reads messages, UTF-16 files and Diffinery's own tokens (#62)
* Live updates carry no event payloads or audit entries, and re-check access every 10 minutes Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016yhgAu6oKfqes812Yd4qjo * Docs: what live updates carry and how long a socket lives Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016yhgAu6oKfqes812Yd4qjo * Push protection reads commit and tag messages, UTF-16 files and Diffinery's job, sandbox and model tokens, and records opt-outs Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016yhgAu6oKfqes812Yd4qjo --------- Co-authored-by: Claude <noreply@anthropic.com>
Canberk Demirci authored · GitHub committed signed1 parent a84e06bcommit 2fd1b061533727489abd8d1f91917a02fc65e420Browse files
9 files changed+166−24
Files (9)
- apps/diffinery/docs/security/audit-log.md+2−0
- apps/diffinery/docs/security/push-protection.md+7−2
- apps/diffinery/docs/self-hosting/architecture.md+1−1
- apps/diffinery/src/git/secret-scan.ts+20−5
- apps/diffinery/src/repo/RepoDO.ts+40−8
- apps/diffinery/src/repo/gitdata.ts+8−3
- apps/diffinery/src/web/assets.ts+3−3
- apps/diffinery/test/e2e/security.test.mjs+57−2
- apps/diffinery/test/unit/secret-scan.test.ts+28−0
modifiedapps/diffinery/docs/security/audit-log.md+2−0View file
| @@ -147,6 +147,8 @@ | ||
| 147 | 147 | | `push.secret_bypassed` | A repository admin pushes past push protection with `git push -o allow-secrets=<reason>`, or a GitHub import imports old history with an audited bypass (see [Imported history](/docs/security/push-protection#imported-history)) | `acme/website` | `kinds`, `reason`, `refs`; for an import also `import` (its id) and `blobs` (up to 50 blob ids), and the reason is `Imported from GitHub <owner>/<name>` | |
| 148 | 148 | | `api.blocked_secret` | [Push protection](/docs/security/push-protection#commits-and-writes-without-git) refuses a commit made with the web editor, `POST /repos/{owner}/{repo}/diffinery/commits`, the contents API or the MCP `commit_files` tool, or a file written with the git data API. The actor is the person or agent who made the request. | `acme/website` | `kinds`, and `paths` (up to 10), or `blob` for the git data API | |
| 149 | 149 | | `api.secret_bypassed` | A repository admin who is a person commits past push protection with `allow_secrets` on `POST /repos/{owner}/{repo}/diffinery/commits` | `acme/website` | `kinds`, `paths`, `reason` | |
| 150 | +| `push.secret_marked` | A push lets through a file that opts out of push protection with `diffinery:allow-secret` and holds something that looks like a credential (see [Opting a file out](/docs/security/push-protection#opting-a-file-out)) | `acme/website` | `kinds`, `blobs` (up to 50 blob ids), `refs` | | |
| 151 | +| `api.secret_marked` | The same, for a commit made without Git or a blob written with the git data API | `acme/website` | `kinds`, and `paths` (up to 10), or `blob` for the git data API | | |
| 150 | 152 | |
| 151 | 153 | ### GitHub imports |
| 152 | 154 |
modifiedapps/diffinery/docs/security/push-protection.md+7−2View file
| @@ -25,6 +25,7 @@ | ||
| 25 | 25 | | `AWS access key` | `AKIA` or `ASIA` followed by 16 uppercase letters or digits | |
| 26 | 26 | | `GitHub token` | `ghp_`, `gho_`, `ghu_`, `ghs_` or `ghr_` followed by 36 letters or digits, or `github_pat_` followed by 82 letters, digits or underscores | |
| 27 | 27 | | `Diffinery token` | `dfy_` followed by at least 32 letters, digits, `_` or `-` (a Diffinery personal or agent token) | |
| 28 | +| `Diffinery job, sandbox or model token` | `djt_`, `dst_` or `dmt_`, a repository id, `_`, and at least 32 more letters, digits, `_` or `-` (a workflow job's `GITHUB_TOKEN`, an agent sandbox's token, or an agent task's model token) | | |
| 28 | 29 | | `Slack token` | `xoxb-`, `xoxa-`, `xoxp-`, `xoxr-` or `xoxs-` followed by at least 10 letters, digits or `-` | |
| 29 | 30 | | `Stripe live key` | `sk_live_` or `rk_live_` followed by at least 20 letters or digits | |
| 30 | 31 | | `Google API key` | `AIza` followed by 35 letters, digits, `_` or `-` | |
| @@ -44,9 +45,11 @@ | ||
| 44 | 45 | |
| 45 | 46 | | Scanned | Not scanned | |
| 46 | 47 | |---|---| |
| 47 | -| Blobs (file contents) contained in the packfile of a `git push` | Commits, trees and tags themselves (commit messages are not scanned) | | |
| 48 | +| Blobs (file contents) contained in the packfile of a `git push` | Trees (file names) | | |
| 49 | +| Commit and annotated tag messages, in a push, a commit made without Git (`message`) and the git data API (`POST .../git/commits`, `POST .../git/tags`) | | | |
| 48 | 50 | | Text files of any size, in a push or a commit made without Git | | |
| 49 | -| Every ref in the push: branches and tags | Binary files: any blob that contains a NUL byte anywhere | | |
| 51 | +| Every ref in the push: branches and tags | Binary files: any blob that contains a NUL byte anywhere, except UTF-16 text | | |
| 52 | +| UTF-16 text files (with a byte order mark) | | | |
| 50 | 53 | | Files you add or change in the web editor | Files that contain the text `diffinery:allow-secret` (see [Opting a file out](#opting-a-file-out)) | |
| 51 | 54 | | Files in a commit made with `POST /repos/{owner}/{repo}/diffinery/commits` or the MCP `commit_files` tool | Objects the server already has, which git does not send again | |
| 52 | 55 | | Files written with `PUT /repos/{owner}/{repo}/contents/{path}` | Git LFS objects, which are uploaded separately from the push | |
| @@ -179,6 +182,8 @@ | ||
| 179 | 182 | |
| 180 | 183 | The opt-out is visible in the code and goes through review like any other change, so use it only for files whose secrets are fake. It matches the exact text `diffinery:allow-secret` (case-sensitive), in any comment syntax. |
| 181 | 184 | |
| 185 | +An opted-out file that holds something that looks like a credential is still let through, and the audit log records it: `push.secret_marked` for a push (with the credential types, up to 50 blob ids and the refs) and `api.secret_marked` for a commit made without Git (with the types and up to 10 paths). So an admin can see who used the opt-out and on what. | |
| 186 | + | |
| 182 | 187 | ## Bypassing push protection |
| 183 | 188 | |
| 184 | 189 | A repository admin who is a person can push a flagged change anyway by giving a reason with a git push option (for commits made through the API, see [Bypassing with the commits API](#bypassing-with-the-commits-api)): |
modifiedapps/diffinery/docs/self-hosting/architecture.md+1−1View file
| @@ -195,7 +195,7 @@ | ||
| 195 | 195 | |
| 196 | 196 | ## Live updates |
| 197 | 197 | |
| 198 | -Pages that show changing state (run logs, job status, pull requests) open a WebSocket to `/<owner>/<repo>/live?topics=...`. The Worker checks that the caller can read the repository, then hands the connection to the repository's Durable Object. The object accepts it with the WebSocket Hibernation API, tagged with up to 10 topics: `repo`, `issues`, `runs`, `run:<id>`, `job:<id>` and `pr:<number>`. Whenever something changes, the object broadcasts a JSON message `{topic, data}` to the sockets tagged with that topic. With hibernation, idle connections don't keep the object in memory. Clients send `ping` and get `pong` to keep the connection open, and reconnect with backoff. | |
| 198 | +Pages that show changing state (run logs, job status, pull requests) open a WebSocket to `/<owner>/<repo>/live?topics=...`. The Worker checks that the caller can read the repository, then hands the connection to the repository's Durable Object. The object accepts it with the WebSocket Hibernation API, tagged with up to 10 topics: `repo`, `issues`, `runs`, `run:<id>`, `job:<id>` and `pr:<number>`. Whenever something changes, the object broadcasts a JSON message `{topic, data}` to the sockets tagged with that topic. With hibernation, idle connections don't keep the object in memory. Clients send `ping` and get `pong` to keep the connection open, and reconnect with backoff. Messages on the `repo` topic only say that something changed (`{kind: "event", type}`, never the event's payload, and never audit entries), and the page fetches what changed with the viewer's own access. Access is checked when a socket connects, so each socket lasts 10 minutes: after that the object closes it with code 4001 and the page reconnects at once, passing the check again. Someone who lost access, or a signed-out reader of a repository made private, stops getting updates within 10 minutes. | |
| 199 | 199 | |
| 200 | 200 | ## Email Sending and Access |
| 201 | 201 |
modifiedapps/diffinery/src/git/secret-scan.ts+20−5View file
| @@ -5,6 +5,8 @@ | ||
| 5 | 5 | { kind: "AWS access key", re: /\b(AKIA|ASIA)[0-9A-Z]{16}\b/ }, |
| 6 | 6 | { kind: "GitHub token", re: /\b(ghp|gho|ghu|ghs|ghr)_[A-Za-z0-9]{36}\b|\bgithub_pat_[A-Za-z0-9_]{82}\b/ }, |
| 7 | 7 | { kind: "Diffinery token", re: /\bdfy_[A-Za-z0-9_-]{32,}\b/ }, |
| 8 | + // A workflow job's, an agent sandbox's or an agent task's model token: dxt_<repository>_..._<secret>. | |
| 9 | + { kind: "Diffinery job, sandbox or model token", re: /\bd(?:jt|st|mt)_[A-Za-z0-9-]{8,64}_[A-Za-z0-9_-]{32,}/ }, | |
| 8 | 10 | { kind: "Slack token", re: /\bxox[baprs]-[A-Za-z0-9-]{10,}\b/ }, |
| 9 | 11 | { kind: "Stripe live key", re: /\b[rs]k_live_[A-Za-z0-9]{20,}\b/ }, |
| 10 | 12 | { kind: "Google API key", re: /\bAIza[0-9A-Za-z_-]{35}\b/ }, |
| @@ -18,13 +20,26 @@ | ||
| 18 | 20 | kind: string; |
| 19 | 21 | } |
| 20 | 22 | |
| 21 | -export function scanForSecrets(blobs: Array<{ oid: string; content: Uint8Array }>): SecretHit[] { | |
| 23 | +/** The text of a blob, or null for a binary one. UTF-16 text (with its byte order mark) is text too. */ | |
| 24 | +function textOf(content: Uint8Array): string | null { | |
| 25 | + if (content[0] === 0xff && content[1] === 0xfe) return new TextDecoder("utf-16le").decode(content); | |
| 26 | + if (content[0] === 0xfe && content[1] === 0xff) return new TextDecoder("utf-16be").decode(content); | |
| 27 | + if (content.includes(0)) return null; | |
| 28 | + return fromUtf8(content); | |
| 29 | +} | |
| 30 | + | |
| 31 | +/** | |
| 32 | + * Credentials in `blobs` (file contents, and commit and tag objects, whose messages are scanned | |
| 33 | + * too). A file marked `diffinery:allow-secret` is let through, but what it holds goes to `marked` | |
| 34 | + * so the caller can put the opt-out on the audit record. | |
| 35 | + */ | |
| 36 | +export function scanForSecrets(blobs: Array<{ oid: string; content: Uint8Array }>, marked: SecretHit[] = []): SecretHit[] { | |
| 22 | 37 | const hits: SecretHit[] = []; |
| 23 | 38 | for (const b of blobs) { |
| 24 | - if (b.content.includes(0)) continue; // binary | |
| 25 | - const text = fromUtf8(b.content); | |
| 26 | - if (/diffinery:allow-secret/.test(text)) continue; // explicit, auditable opt-out for test fixtures | |
| 27 | - for (const p of PATTERNS) if (p.re.test(text)) hits.push({ oid: b.oid, kind: p.kind }); | |
| 39 | + const text = textOf(b.content); | |
| 40 | + if (text === null) continue; | |
| 41 | + const out = /diffinery:allow-secret/.test(text) ? marked : hits; | |
| 42 | + for (const p of PATTERNS) if (p.re.test(text)) out.push({ oid: b.oid, kind: p.kind }); | |
| 28 | 43 | } |
| 29 | 44 | return hits; |
| 30 | 45 | } |
modifiedapps/diffinery/src/repo/RepoDO.ts+40−8View file
| @@ -26,6 +26,7 @@ | ||
| 26 | 26 | type Commit, |
| 27 | 27 | type FileChange, |
| 28 | 28 | type Signature, |
| 29 | + utf8, | |
| 29 | 30 | } from "@diffinery/git-core"; |
| 30 | 31 | import type { DiffineryEvent } from "@diffinery/shared"; |
| 31 | 32 | import type { Env } from "../env.js"; |
| @@ -39,7 +40,7 @@ | ||
| 39 | 40 | type RefEntry, |
| 40 | 41 | type RefResult, |
| 41 | 42 | } from "../git/protocol.js"; |
| 42 | -import { scanForSecrets, secretBypassReason } from "../git/secret-scan.js"; | |
| 43 | +import { scanForSecrets, secretBypassReason, type SecretHit } from "../git/secret-scan.js"; | |
| 43 | 44 | import { HttpError, isValidRefName, now } from "../lib/util.js"; |
| 44 | 45 | import { deletePrefixes, transientPrefixes } from "../lib/storage-keys.js"; |
| 45 | 46 | import { Actions } from "./actions.js"; |
| @@ -87,6 +88,11 @@ | ||
| 87 | 88 | * refs, Git objects, issues, pull requests, reviews, checks and workflow runs, |
| 88 | 89 | * which gives strongly consistent ref updates and merges without locks. |
| 89 | 90 | */ |
| 91 | +/** How long a live-update socket lives before the page must reconnect and pass the access check again. */ | |
| 92 | +const LIVE_REAUTH_MS = 10 * 60 * 1000; | |
| 93 | +/** The close code that asks a page to reconnect at once (not a failure: no backoff, no reload). */ | |
| 94 | +export const LIVE_REAUTH_CODE = 4001; | |
| 95 | + | |
| 90 | 96 | export class RepoDO extends DurableObject<Env> { |
| 91 | 97 | readonly sql: SqlStorage; |
| 92 | 98 | store!: RepoObjectStore; |
| @@ -201,7 +207,9 @@ | ||
| 201 | 207 | /** Queue an event (webhooks, notifications) and show it live to open pages; `live: false` skips the pages (e.g. draft releases, which readers can't see). */ |
| 202 | 208 | emit(event: DiffineryEvent, opts?: { live?: boolean }): void { |
| 203 | 209 | this.ctx.waitUntil(this.env.EVENTS.send(event).catch((e) => console.error("event send failed", e))); |
| 204 | - if (opts?.live !== false) this.broadcast("repo", { kind: "event", event }); | |
| 210 | + // Open pages only learn that something changed and fetch it with the viewer's own access: the event | |
| 211 | + // itself (an audit entry's detail, a dispatch's payload) is not every reader's to see. | |
| 212 | + if (opts?.live !== false && event.type !== "audit") this.broadcast("repo", { kind: "event", type: event.type }); | |
| 205 | 213 | } |
| 206 | 214 | |
| 207 | 215 | // ---------------------------------------------------------------- refs |
| @@ -310,14 +318,18 @@ | ||
| 310 | 318 | unpackError = (e as Error).message; |
| 311 | 319 | } |
| 312 | 320 | } |
| 313 | - const newBlobs = objects.filter((o) => o.type === "blob").map((o) => ({ oid: o.oid, content: o.content })); | |
| 321 | + // Commit and tag messages are scanned too, not only files. | |
| 322 | + const newBlobs = objects.filter((o) => o.type === "blob" || o.type === "commit" || o.type === "tag").map((o) => ({ oid: o.oid, content: o.content })); | |
| 314 | 323 | const results: RefResult[] = []; |
| 315 | 324 | if (unpackError) { |
| 316 | 325 | for (const c of req.commands) results.push({ ref: c.ref, ok: false, reason: "unpacker error" }); |
| 317 | 326 | return reportStatus(req, unpackError, results); |
| 318 | 327 | } |
| 319 | 328 | |
| 320 | - const leaks = scanForSecrets(newBlobs); | |
| 329 | + const marked: SecretHit[] = []; | |
| 330 | + const leaks = scanForSecrets(newBlobs, marked); | |
| 331 | + // A file can opt out with `diffinery:allow-secret`; that it did is on the record. | |
| 332 | + if (marked.length) this.emit({ type: "audit", action: "push.secret_marked", actor: caller, target: `${this.meta.owner}/${this.meta.name}`, detail: { kinds: [...new Set(marked.map((l) => l.kind))], blobs: marked.slice(0, 50).map((l) => l.oid), refs: req.commands.map((c) => c.ref) }, at: now() }); | |
| 321 | 333 | if (leaks.length) { |
| 322 | 334 | const kinds = leaks.map((l) => l.kind); |
| 323 | 335 | const target = `${this.meta.owner}/${this.meta.name}`; |
| @@ -573,13 +585,16 @@ | ||
| 573 | 585 | * Push protection for commits made without a Git client: the same scan as `git push`, so the |
| 574 | 586 | * web editor, the contents and commits APIs and MCP cannot be used to slip a credential in. |
| 575 | 587 | */ |
| 576 | - private protectCommit(changes: readonly FileChange[], allowSecrets: string | undefined, caller: Caller): void { | |
| 588 | + private protectCommit(changes: readonly FileChange[], allowSecrets: string | undefined, caller: Caller, message = ""): void { | |
| 577 | 589 | const blobs = changes.filter((ch) => !ch.delete && ch.content).map((ch) => ({ oid: ch.path, content: ch.content! })); |
| 578 | - const leaks = scanForSecrets(blobs); | |
| 590 | + if (message) blobs.push({ oid: "(commit message)", content: utf8(message) }); | |
| 591 | + const marked: SecretHit[] = []; | |
| 592 | + const leaks = scanForSecrets(blobs, marked); | |
| 593 | + const target = `${this.meta.owner}/${this.meta.name}`; | |
| 594 | + if (marked.length) this.emit({ type: "audit", action: "api.secret_marked", actor: caller, target, detail: { kinds: [...new Set(marked.map((l) => l.kind))], paths: [...new Set(marked.map((l) => l.oid))].slice(0, 10) }, at: now() }); | |
| 579 | 595 | if (!leaks.length) return; |
| 580 | 596 | const kinds = [...new Set(leaks.map((l) => l.kind))]; |
| 581 | 597 | const paths = [...new Set(leaks.map((l) => l.oid))].slice(0, 10); |
| 582 | - const target = `${this.meta.owner}/${this.meta.name}`; | |
| 583 | 598 | const reason = allowSecrets?.trim().slice(0, 200); |
| 584 | 599 | if (reason && caller.role === "admin" && caller.kind === "user") { |
| 585 | 600 | this.emit({ type: "audit", action: "api.secret_bypassed", actor: caller, target, detail: { kinds, paths, reason }, at: now() }); |
| @@ -616,7 +631,7 @@ | ||
| 616 | 631 | const why = ch.delete ? null : unsafeTreePath(ch.path); |
| 617 | 632 | if (why) throw new HttpError(422, `invalid path ${JSON.stringify(ch.path)}: ${why}`); |
| 618 | 633 | } |
| 619 | - this.protectCommit(input.changes, input.allowSecrets, caller); | |
| 634 | + this.protectCommit(input.changes, input.allowSecrets, caller, input.message); | |
| 620 | 635 | const baseTree = parent ? (await readCommit(this.store, parent)).tree : null; |
| 621 | 636 | const tree = await applyChanges(this.store, baseTree, input.changes); |
| 622 | 637 | const ts = Math.floor(Date.now() / 1000); |
| @@ -1519,10 +1534,26 @@ | ||
| 1519 | 1534 | const topics = (new URL(request.url).searchParams.get("topics") ?? "repo").split(",").filter((t) => /^[a-z]+(:[A-Za-z0-9_-]+)?$/.test(t)).slice(0, 10); |
| 1520 | 1535 | const pair = new WebSocketPair(); |
| 1521 | 1536 | this.ctx.acceptWebSocket(pair[1], topics.length ? topics : ["repo"]); |
| 1537 | + // Access is checked when a page connects; reconnecting checks it again, so someone who lost access | |
| 1538 | + // (removed, or the repository made private) stops getting updates within LIVE_REAUTH_MS. | |
| 1539 | + pair[1].serializeAttachment({ until: Date.now() + LIVE_REAUTH_MS }); | |
| 1522 | 1540 | return new Response(null, { status: 101, webSocket: pair[0] }); |
| 1523 | 1541 | } |
| 1524 | 1542 | |
| 1543 | + /** Close a socket whose access check is too old; the page reconnects, and the Worker checks access again. */ | |
| 1544 | + private expired(ws: WebSocket): boolean { | |
| 1545 | + const a = ws.deserializeAttachment() as { until?: number } | null; | |
| 1546 | + if (a?.until && a.until > Date.now()) return false; | |
| 1547 | + try { | |
| 1548 | + ws.close(LIVE_REAUTH_CODE, "reconnect"); | |
| 1549 | + } catch { | |
| 1550 | + /* already closed */ | |
| 1551 | + } | |
| 1552 | + return true; | |
| 1553 | + } | |
| 1554 | + | |
| 1525 | 1555 | override async webSocketMessage(ws: WebSocket, message: string | ArrayBuffer): Promise<void> { |
| 1556 | + if (this.expired(ws)) return; | |
| 1526 | 1557 | if (message === "ping") ws.send("pong"); |
| 1527 | 1558 | } |
| 1528 | 1559 | |
| @@ -1537,6 +1568,7 @@ | ||
| 1537 | 1568 | broadcast(topic: string, data: unknown): void { |
| 1538 | 1569 | const msg = JSON.stringify({ topic, data }); |
| 1539 | 1570 | for (const ws of this.ctx.getWebSockets(topic)) { |
| 1571 | + if (this.expired(ws)) continue; | |
| 1540 | 1572 | try { |
| 1541 | 1573 | ws.send(msg); |
| 1542 | 1574 | } catch { |
modifiedapps/diffinery/src/repo/gitdata.ts+8−3View file
| @@ -18,9 +18,10 @@ | ||
| 18 | 18 | type ObjectType, |
| 19 | 19 | type Signature, |
| 20 | 20 | type TreeEntry, |
| 21 | + utf8, | |
| 21 | 22 | } from "@diffinery/git-core"; |
| 22 | 23 | import type { Env } from "../env.js"; |
| 23 | -import { scanForSecrets } from "../git/secret-scan.js"; | |
| 24 | +import { scanForSecrets, type SecretHit } from "../git/secret-scan.js"; | |
| 24 | 25 | import { ZipWriter, tarEnd, tarEntry, tarStart, zip, type ArchiveEntry } from "../lib/archive.js"; |
| 25 | 26 | import { R2Sink } from "../lib/r2-sink.js"; |
| 26 | 27 | import { unsafeTreeName, unsafeTreePath } from "../lib/tree-names.js"; |
| @@ -107,9 +108,11 @@ | ||
| 107 | 108 | |
| 108 | 109 | /** Same push protection as git pushes: blobs with credentials are refused. */ |
| 109 | 110 | private scan(oid: string, content: Uint8Array, caller: Caller): void { |
| 110 | - const hits = scanForSecrets([{ oid, content }]); | |
| 111 | + const marked: SecretHit[] = []; | |
| 112 | + const hits = scanForSecrets([{ oid, content }], marked); | |
| 113 | + const m = this.repo.meta; | |
| 114 | + if (marked.length) this.repo.emit({ type: "audit", action: "api.secret_marked", actor: caller, target: `${m.owner}/${m.name}`, detail: { kinds: marked.map((h) => h.kind), blob: oid }, at: now() }); | |
| 111 | 115 | if (hits.length) { |
| 112 | - const m = this.repo.meta; | |
| 113 | 116 | this.repo.emit({ type: "audit", action: "api.blocked_secret", actor: caller, target: `${m.owner}/${m.name}`, detail: { kinds: hits.map((h) => h.kind), blob: oid }, at: now() }); |
| 114 | 117 | throw new HttpError(422, `push protection: possible secret detected (${hits.map((h) => h.kind).join(", ")})`); |
| 115 | 118 | } |
| @@ -247,6 +250,7 @@ | ||
| 247 | 250 | const parents = input.parents ?? []; |
| 248 | 251 | if (!Array.isArray(parents)) throw new HttpError(422, "parents must be an array"); |
| 249 | 252 | for (const p of parents) if (!isOid(String(p)) || this.objectType(String(p)) !== "commit") throw new HttpError(422, `parent ${p} is not a valid commit`, { errors: [{ resource: "Commit", code: "invalid", field: "parents" }] }); |
| 253 | + this.scan("(commit message)", utf8(input.message), caller); | |
| 250 | 254 | const me = this.signature(caller); |
| 251 | 255 | const author = parseSig(input.author, me, "author"); |
| 252 | 256 | const committer = parseSig(input.committer, input.author ? author : me, "committer"); |
| @@ -273,6 +277,7 @@ | ||
| 273 | 277 | if (type !== actual) throw new HttpError(422, `object is a ${actual}, not a ${type}`, { errors: [{ resource: "Tag", code: "invalid", field: "type" }] }); |
| 274 | 278 | const tagger = parseSig(input.tagger, this.signature(caller), "tagger"); |
| 275 | 279 | const message = String(input.message ?? ""); |
| 280 | + this.scan("(tag message)", utf8(message), caller); | |
| 276 | 281 | const sha = await this.store.write("tag", serializeTag({ object, type, tag, tagger, message: message.endsWith("\n") || !message ? message : `${message}\n` })); |
| 277 | 282 | return this.getTag(sha); |
| 278 | 283 | } |
modifiedapps/diffinery/src/web/assets.ts+3−3View file
| @@ -441,11 +441,11 @@ | ||
| 441 | 441 | }); |
| 442 | 442 | // Live updates |
| 443 | 443 | const live=b.dataset.live,mode=b.dataset.liveMode||'reload'; |
| 444 | -if(live&&repo&&'WebSocket' in window){let ws,tries=0,timer=null,lastSeq=Number(b.dataset.lastSeq||'-1'),opened=false; | |
| 444 | +if(live&&repo&&'WebSocket' in window){let ws,tries=0,timer=null,lastSeq=Number(b.dataset.lastSeq||'-1'),opened=false,planned=false; | |
| 445 | 445 | function connect(){ws=new WebSocket((location.protocol==='https:'?'wss://':'ws://')+location.host+'/'+repo+'/live?topics='+encodeURIComponent(live)); |
| 446 | - ws.onopen=()=>{if(opened){if(mode==='log')location.reload();else refresh()}opened=true;tries=0}; | |
| 446 | + ws.onopen=()=>{if(opened){if(mode!=='log')refresh();else if(!planned)location.reload()}opened=true;planned=false;tries=0}; | |
| 447 | 447 | ws.onmessage=e=>{if(e.data==='pong')return;let m;try{m=JSON.parse(e.data)}catch(_){return}handle(m)}; |
| 448 | - ws.onclose=()=>{setTimeout(connect,Math.min(30000,1000*Math.pow(2,tries++)))}} | |
| 448 | + ws.onclose=e=>{planned=e.code===4001;setTimeout(connect,planned?0:Math.min(30000,1000*Math.pow(2,tries++)))}} | |
| 449 | 449 | setInterval(()=>{if(ws&&ws.readyState===1)ws.send('ping')},25000); |
| 450 | 450 | function handle(m){const a=m.data&&m.data.action; |
| 451 | 451 | if(mode==='log'){if(!String(m.topic).startsWith('job:'))return; |
modifiedapps/diffinery/test/e2e/security.test.mjs+57−2View file
| @@ -359,10 +359,14 @@ | ||
| 359 | 359 | ws.addEventListener("error", rej); |
| 360 | 360 | }); |
| 361 | 361 | const draft = await ok("POST", `/repos/${OWNER}/flip/releases`, { tag_name: "v0.9", name: "Secret plans", body: "draft notes", draft: true }); |
| 362 | + // Live messages only say what kind of thing changed: nothing for the draft, then the published release. | |
| 363 | + const releases = () => seen.filter((m) => JSON.parse(m).data?.type === "release"); | |
| 364 | + await sleep(700); | |
| 365 | + assert.equal(releases().length, 0, seen.join("\n")); | |
| 362 | 366 | await ok("POST", `/repos/${OWNER}/flip/releases`, { tag_name: "v1.0", name: "Public release" }); |
| 363 | - await until("the published release event", async () => seen.some((m) => m.includes("Public release"))); | |
| 367 | + await until("the published release event", async () => releases().length > 0); | |
| 364 | 368 | ws.close(); |
| 365 | - assert.ok(!seen.some((m) => m.includes("Secret plans") || m.includes(`"id":${draft.id}`)), `draft leaked: ${seen.join("\n")}`); | |
| 369 | + assert.ok(!seen.some((m) => m.includes("Secret plans") || m.includes("Public release") || m.includes(`"id":${draft.id}`)), `release details leaked: ${seen.join("\n")}`); | |
| 366 | 370 | }); |
| 367 | 371 | |
| 368 | 372 | test("security: organization admins can't hand out ownership", async () => { |
| @@ -1496,3 +1500,54 @@ | ||
| 1496 | 1500 | assert.equal(hidden.page[0], 302); |
| 1497 | 1501 | assert.match(hidden.page[1], /\/login\?next=/); |
| 1498 | 1502 | }); |
| 1503 | + | |
| 1504 | +test("security: live updates say only that something changed, and never carry audit entries or payloads", async () => { | |
| 1505 | + const R = `/repos/${OWNER}/on-air`; | |
| 1506 | + await repoWithReadme("on-air", { private: false }); | |
| 1507 | + // A signed-out reader watching the public repository. | |
| 1508 | + const ws = new WebSocket(`${base.replace("http", "ws")}/${OWNER}/on-air/live?topics=repo`); | |
| 1509 | + const got = []; | |
| 1510 | + ws.onmessage = (e) => e.data !== "pong" && got.push(JSON.parse(e.data)); | |
| 1511 | + await new Promise((ok, fail) => ((ws.onopen = ok), (ws.onerror = fail))); | |
| 1512 | + // A dispatch with a payload, and a commit refused by push protection (an audit entry naming the path). | |
| 1513 | + await status(204, "POST", `${R}/dispatches`, { event_type: "deploy", client_payload: { note: "for writers only" } }); | |
| 1514 | + await status(422, "POST", `${R}/diffinery/commits`, { branch: "main", message: "leak", files: [{ path: "keys/prod.env", content: `AWS_ACCESS_KEY_ID=${"AKIA"}IOSFODNN7EXAMPL3\n` }] }); | |
| 1515 | + await until("the dispatch on the socket", async () => got.some((m) => m.data?.type === "repository_dispatch")); | |
| 1516 | + await sleep(500); | |
| 1517 | + ws.close(); | |
| 1518 | + const text = JSON.stringify(got); | |
| 1519 | + assert.doesNotMatch(text, /for writers only|keys\/prod\.env|audit|blocked_secret/); | |
| 1520 | + for (const m of got) assert.deepEqual(Object.keys(m.data).sort(), ["kind", "type"]); | |
| 1521 | +}); | |
| 1522 | + | |
| 1523 | +test("security: push protection reads commit and tag messages, UTF-16 files and Diffinery's own tokens, and records opt-outs", async () => { | |
| 1524 | + const R = `/repos/${OWNER}/messages`; | |
| 1525 | + const mainSha = await repoWithReadme("messages"); | |
| 1526 | + const key = `AWS_ACCESS_KEY_ID=${"AKIA"}IOSFODNN7EXAMPL3`; | |
| 1527 | + // A commit message, through git. | |
| 1528 | + const clone = join(work, "messages"); | |
| 1529 | + execFileSync("git", ["clone", "-q", `${base.replace("://", `://x:${T.olive}@`)}/${OWNER}/messages.git`, clone], { stdio: "pipe" }); | |
| 1530 | + const g = (...args) => execFileSync("git", ["-c", "user.name=Olive", "-c", "user.email=olive@example.com", ...args], { cwd: clone, stdio: "pipe" }); | |
| 1531 | + g("commit", "-q", "--allow-empty", "-m", `deploy\n\n${key}`); | |
| 1532 | + assert.throws(() => g("push", "-q", "origin", "HEAD:main"), /secret detected/); | |
| 1533 | + // ...an annotated tag's message. | |
| 1534 | + g("reset", "-q", "--hard", "origin/main"); | |
| 1535 | + g("tag", "-a", "v1", "-m", key); | |
| 1536 | + assert.throws(() => g("push", "-q", "origin", "v1"), /secret detected/); | |
| 1537 | + // ...and without git: the commits API and the git data API. | |
| 1538 | + await status(422, "POST", `${R}/diffinery/commits`, { branch: "main", message: `notes\n${key}`, files: [{ path: "a.txt", content: "a\n" }] }); | |
| 1539 | + const tree = (await ok("GET", `${R}/git/commits/${mainSha}`)).tree.sha; | |
| 1540 | + await status(422, "POST", `${R}/git/commits`, { message: key, tree, parents: [mainSha] }); | |
| 1541 | + await status(422, "POST", `${R}/git/tags`, { tag: "v2", message: key, object: mainSha, type: "commit" }); | |
| 1542 | + // A UTF-16 file has NUL bytes but is text. | |
| 1543 | + const utf16 = Buffer.concat([Buffer.from([0xff, 0xfe]), Buffer.from(`${key}\n`, "utf16le")]); | |
| 1544 | + await status(422, "POST", `${R}/diffinery/commits`, { branch: "main", message: "utf16", files: [{ path: "creds.txt", content: utf16.toString("base64"), encoding: "base64" }] }); | |
| 1545 | + // A job, sandbox or model token. | |
| 1546 | + const token = `dmt_${"0b1e5c4e-58a0-4f7c-9d39-0e2c6a1c3f00"}_${"Zm9vYmFyYmF6cXV4cXV1eHF1dXhxdXV4cXV1eA"}`; | |
| 1547 | + await status(422, "POST", `${R}/diffinery/commits`, { branch: "main", message: "token", files: [{ path: "token.txt", content: token }] }); | |
| 1548 | + assert.equal((await ok("GET", `${R}/branches/main`)).commit.sha, mainSha); | |
| 1549 | + // A file that opts out still goes in, and the opt-out is on the record. | |
| 1550 | + await ok("POST", `${R}/diffinery/commits`, { branch: "main", message: "fixture", files: [{ path: "fixture.env", content: `${key} # ${"diffinery"}:allow-secret\n` }] }); | |
| 1551 | + const entry = await until("the opt-out in the audit log", async () => auditLog("api.secret_marked").find((e) => e.target === `${OWNER}/messages`)); | |
| 1552 | + assert.deepEqual(JSON.parse(entry.detail).paths, ["fixture.env"]); | |
| 1553 | +}); |
modifiedapps/diffinery/test/unit/secret-scan.test.ts+28−0View file
| @@ -10,6 +10,34 @@ | ||
| 10 | 10 | expect(scanForSecrets([blob("key = AKIAIOSFODNN7EXAMPL3 # diffinery:allow-secret")])).toEqual([]); |
| 11 | 11 | }); |
| 12 | 12 | |
| 13 | + it("records what an opted-out file holds", () => { | |
| 14 | + const marked: { oid: string; kind: string }[] = []; | |
| 15 | + expect(scanForSecrets([blob("key = AKIAIOSFODNN7EXAMPL3 # diffinery:allow-secret")], marked)).toEqual([]); | |
| 16 | + expect(marked.map((h) => h.kind)).toEqual(["AWS access key"]); | |
| 17 | + }); | |
| 18 | + | |
| 19 | + it("flags Diffinery's job, sandbox and model tokens", () => { | |
| 20 | + const repo = "0b1e5c4e-58a0-4f7c-9d39-0e2c6a1c3f00"; | |
| 21 | + const secret = "Zm9vYmFyYmF6cXV4cXV1eHF1dXhxdXV4cXV1eA"; | |
| 22 | + for (const t of [`djt_${repo}_17_${secret}`, `dst_${repo}_sb1_${secret}`, `dmt_${repo}_${secret}`]) { | |
| 23 | + expect(scanForSecrets([blob(`TOKEN=${t}`)]).map((h) => h.kind), t).toEqual(["Diffinery job, sandbox or model token"]); | |
| 24 | + } | |
| 25 | + expect(scanForSecrets([blob("see dst_ and dmt_ prefixes in the docs")])).toEqual([]); | |
| 26 | + }); | |
| 27 | + | |
| 28 | + it("reads UTF-16 files, which have NUL bytes but are text", () => { | |
| 29 | + const text = "AWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPL3\n"; | |
| 30 | + const le = new Uint8Array(2 + text.length * 2); | |
| 31 | + le.set([0xff, 0xfe]); | |
| 32 | + for (let i = 0; i < text.length; i++) le[2 + i * 2] = text.charCodeAt(i); | |
| 33 | + const be = new Uint8Array(le.length); | |
| 34 | + be.set([0xfe, 0xff]); | |
| 35 | + for (let i = 0; i < text.length; i++) be[3 + i * 2] = text.charCodeAt(i); | |
| 36 | + for (const content of [le, be]) expect(scanForSecrets([{ oid: "a".repeat(40), content }]).map((h) => h.kind)).toEqual(["AWS access key"]); | |
| 37 | + // Other files with NUL bytes are binary and skipped. | |
| 38 | + expect(scanForSecrets([{ oid: "b".repeat(40), content: new Uint8Array([0x41, 0, 0x41]) }])).toEqual([]); | |
| 39 | + }); | |
| 40 | + | |
| 13 | 41 | it("reads the bypass reason from push options", () => { |
| 14 | 42 | expect(secretBypassReason([])).toBeNull(); |
| 15 | 43 | expect(secretBypassReason(["allow-secrets="])).toBeNull(); |