Security and docs-review fixes, Actions runtime hardening, and needs-graph job scheduling (#29)
* Serve GitHub's Actions runtime services: cache, artifacts and signed blob URLs
The official toolkit (actions/cache, upload-artifact, download-artifact,
buildkit's type=gha cache) now talks to Diffinery as it talks to GitHub:
- Twirp CacheService (cache v2) and ArtifactService (artifacts v4), plus the
legacy cache REST API, authenticated by a per-job ACTIONS_RUNTIME_TOKEN
(HS256 JWT under a key derived from MASTER_KEY) that only works while the
job runs and expires with its timeout.
- Cache semantics like GitHub's: key, restore keys and version; lookups in the
job's ref, then the PR base, then the default branch; saves to the own ref;
10 GB per repository with LRU and 7-day idle eviction.
- Artifacts with numeric ids, per-run names, overwrite via DeleteArtifact,
retention up to 90 days, digests, and unarchived single files.
- Signed, expiring, blob-scoped URLs that speak Put Block, Put Block List,
Put Blob and ranged GET; blocks are staged in R2 and composed into one
object with a multipart upload.
- REST: artifacts (list, get, delete, zip download), run artifacts, caches
(list, delete by key or id, usage).
- github.run_id is the numeric run number; the API accepts it as a run id.
- Event payloads get GitHub's full repository, sender, organization and
pull_request objects (head.repo, base.repo).
- The job token's role follows the workflow's permissions (read by default),
and what a job does with it starts no new runs, as on GitHub.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AizKXSsengXZRegJxRATkL
* Runner: speak the Actions runtime services, and run real actions
The runner now exports ACTIONS_RUNTIME_TOKEN, ACTIONS_RESULTS_URL,
ACTIONS_CACHE_URL, ACTIONS_RUNTIME_URL and ACTIONS_CACHE_SERVICE_V2, so
the official cache and artifact actions work unmodified. The built-in
cache and artifact actions use the same Twirp services, archive formats
and cache versions, so entries and artifacts are shared both ways.
Built-ins now step aside for the real action when the ref is not a
known major, an input is unknown, or DIFFINERY_BUILTINS is false in the
workflow (DIFFINERY_DISABLE_BUILTINS=1 on the runner). github-script is
no longer refused. Artifact actions see GITHUB_SERVER_URL=github.com
(@actions/artifact refuses GHES) and their artifact-url is mapped back.
Action downloads fall back to git when the tarball endpoint is blocked.
setup-node/python/go cache with the official actions' keys, and gain
always-auth, allow-prereleases, pipenv and npm auto-caching. ImageOS,
ImageVersion and ACTIONS_RUNNER_DEBUG are set like hosted runners.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AizKXSsengXZRegJxRATkL
* Runner: checkout parity for LFS, sparse-checkout, submodules and filters
The checkout built-in now supports lfs, sparse-checkout (cone and
non-cone), filter, show-progress, set-safe-directory and
github-server-url. Credentials are scoped to the server's origin, live
in the repository's config during the fetch (so git-lfs and submodule
clones use them) and are removed right away when persist-credentials is
false. The job's own token is never sent to another host.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AizKXSsengXZRegJxRATkL
* Runner image: zstd, gh and ImageOS like hosted ubuntu-24.04
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AizKXSsengXZRegJxRATkL
* Accept a repeated Authorization header, and artifact URLs by id
git-lfs sends http.extraheader and the batch API's header together, which
arrived joined and failed to decode (a 500 on every LFS upload with an
extra header). Malformed basic credentials are now a 401. The artifact
download page also accepts the artifact id that upload-artifact links to.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AizKXSsengXZRegJxRATkL
* GITHUB_TOKEN follows GitHub's permissions: model, per endpoint
A job token's permissions come from the job's `permissions:`, else the
workflow's, else the repository default (write, GitHub's permissive
default). read-all, write-all, {} and per-scope maps resolve like
GitHub; reusable workflows never exceed their caller, and pull requests
from another repository only read.
loadRepo checks each request against the scope its endpoint needs
(issues, pull-requests, contents, statuses, checks, deployments,
actions, ...), answering 403 "Resource not accessible by integration".
Repository settings, secrets, variables, hooks, runners and branch
protection stay out of reach, the token's role never exceeds write so
branch protection applies, and runs started by an agent are held to
the agent rules. OIDC and the container registry's packages permission
use the same effective permissions.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AizKXSsengXZRegJxRATkL
* Job and sandbox tokens can read public repositories
Like GITHUB_TOKEN: a workflow can clone a public submodule or fetch a
public repository's artifacts, but not another private repository.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AizKXSsengXZRegJxRATkL
* Artifact downloads may name the job that uploaded
download-artifact sends the artifact's own run and job ids when it asks
for a download URL, so reads only need to name the token's run.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AizKXSsengXZRegJxRATkL
* E2E with the official actions; document what works in workflows
Two runs of a workflow using the real actions/checkout, cache@v4 and
@v3, upload/download-artifact@v4, github-script@v7, setup-node@v4 (with
DIFFINERY_DISABLE_BUILTINS=1) and setup-java@v4 next to the built-ins:
LFS, submodules over HTTP and sparse checkout; cache misses then hits
across the real actions and the built-ins; artifacts across jobs and,
with run-id and github-token, across runs; github-script creating a
label, issue, comment and commit status with the default token; and a
job with `permissions: { contents: read }` refused an issue write.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AizKXSsengXZRegJxRATkL
* Job tokens upload release assets and write commit comments with contents
Release asset uploads arrive at /api/uploads/repos/..., and commit comments
live under /comments and /commits/:sha/comments; both need contents, as on
GitHub.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AizKXSsengXZRegJxRATkL
* Registry e2e: jobs that check out ask for contents: read
Naming any permission sets the others to none, so a job with only
packages: write cannot fetch a private repository, here as on GitHub.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AizKXSsengXZRegJxRATkL
* Live test: read the webhook's delivery from GitHub's hook shape
The hooks API now returns GitHub's shape (#19): the target is config.url
and the last delivery's status is last_response.code. The live test still
read lastDeliveryAt, so it waited for a field that no longer exists.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AizKXSsengXZRegJxRATkL
* Runner: keep the job token out of git config and run steps
- checkout passes credentials to git through GIT_CONFIG_* environment while
fetching and writes them to .git/config only at the end, when
persist-credentials is on, so a failed checkout leaves nothing behind;
stale extraheaders from a reused workspace are removed first.
- ACTIONS_RUNTIME_TOKEN, ACTIONS_RESULTS_URL and ACTIONS_CACHE_URL go to
JavaScript and Docker actions and the built-ins, not to `run:` steps
(as on GitHub's runner).
- DIFFINERY_GITHUB_TOKEN is read once at startup, never reaches steps, and is
passed to git for action clones through the environment, not argv.
- A test pins that gzip is part of the cache version (as in @actions/cache).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AizKXSsengXZRegJxRATkL
* Job token permissions: restricted default, web paths, public reads
- RESTRICTED_DEFAULT (contents and packages read) and a repoDefault option,
GitHub's "Read repository contents and packages permissions" setting.
- Web paths: the repository root and /commit/, /edit/, /new/ need contents;
/pull/ needs pull-requests. The root was refused outright before (the
optional path group came back undefined).
- A request that only reads a public repository is allowed whatever the
token's permissions (contents: none still reads its own public repository,
as anyone could).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AizKXSsengXZRegJxRATkL
* Actions storage: own R2 prefixes, upload budgets, scalable compose
Storage layout (the bootstrap lifecycle rule deleted everything under repos/
after 90 days, git objects included):
- caches under actions-cache/<repo>/ (no age rule; Diffinery evicts),
artifacts under artifacts/<repo>/ and logs under logs/<repo>/ (90 days),
uploads in progress under actions-staging/ (1 day). repos/ has no age
rule; unfinished multipart uploads are aborted after a day everywhere.
- Objects written under the old repos/<id>/{cache,artifacts,logs}/ keys are
still read. A cache row whose object is gone is dropped and the lookup
retried.
- bootstrap.sh fails the deploy when the lifecycle rules can't be applied
or would still delete anything under repos/.
Upload limits:
- Put Block, Put Blob and the v1 PATCH check that the upload is still
writable by this job, cap a request at 256 MiB and count staged bytes
against the budgets: 10 GB of caches per repository (LRU eviction) and an
artifact quota (ARTIFACT_QUOTA_BYTES, 10 GB by default). Commits over the
limit are refused.
- R2 objects are deleted whenever their rows are (eviction, replacement,
expiry, deletion), and completeJob sweeps the job's unfinished uploads.
- Bodies stream into R2: with Content-Length (and the v1 PATCH's
Content-Range) through a FixedLengthStream, otherwise a part at a time.
actions/cache takes the v1 path here (32 MB chunks, 4 at a time, no
Content-Length), so nothing is buffered whole.
- The block list's size is checked before it is read, and the total before
composing.
Committed entries can't be replaced: the object's etag is recorded at
finalize, and the blob is served only while it still matches.
Compose scales to 50,000 blocks: small blocks are merged in rounds and the
parts are uploaded in batches, each step in its own Durable Object call, so no
invocation makes more than 400 subrequests. Unit tests count the calls.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AizKXSsengXZRegJxRATkL
* Workflow tokens: repository settings, no approvals or workflow edits, agent origin
- Repository Actions settings, GitHub's "Workflow permissions":
GET/PUT /repos/{owner}/{repo}/actions/permissions/workflow
(default_workflow_permissions, can_approve_pull_request_reviews). New
repositories default to read-only tokens (contents and packages read);
repositories created before keep write. Only people with admin can change
it, and each change is audited.
- Job tokens can't approve, request changes on or open pull requests unless
the repository allows it (GitHub's message). Only people's approvals count
toward required approvals.
- Job tokens never create, change or delete files under .github/workflows/
or .diffinery/workflows/, by push, API commit or merge: "refusing to allow a
GitHub App to create or update workflow `<path>` without `workflows`
permission".
- A run whose code came from an agent (an agent triggered it, pushed the
branch last, or opened the pull request) is marked agent_origin; re-runs
keep it whoever starts them. Its jobs act as an agent, so the agent
guardrails (no pushes or merges to default or protected branches) hold.
ref_pushers records who last moved each ref.
- Whether a pull_request run is from a fork comes from the pull request's row
(pulls.head_repo_id), not the event payload.
- E2E: the default token is read-only, workflow pushes are refused, agent-
pushed code can't reach main through a job token, approvals are refused
until allowed and then don't count. The toolkit e2e opts its repository
into write, as older repositories are.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AizKXSsengXZRegJxRATkL
* Warm the TypeScript grammar before asserting on its tokens
The first tokenization compiles the grammar's regexes. On a busy CI runner
that alone can pass shiki's 200 ms tokenize limit, which leaves the rest of
the line as one plain token, so 'const' was never a token of its own and
the test failed on main since #23.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AizKXSsengXZRegJxRATkL
* Close API holes the docs writers found
- Anonymous callers can no longer create issues, comments or reactions on
public repositories; anonymous git fetches still work.
- Token scopes are enforced: a read-scoped token only reads, account-level
writes need write scope, and orgs, invitations, agents and tokens need
admin scope. Scopes are validated on the server.
- Agents can't add, change or delete secrets or variables at any level.
- A merge commit is refused on branches that require linear history, and
admins bypass unmet rules only with an explicit, audited bypass_rules.
- Drafts can be marked ready for review (PATCH draft, web button, CLI
`pr ready`).
- Failed webhook deliveries are retried with backoff; workflow_dispatch and
edited events are emitted.
- The exact branch protection rule wins over patterns, then the oldest.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AizKXSsengXZRegJxRATkL
* Account-level changes refuse tokens limited to repositories
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AizKXSsengXZRegJxRATkL
* Close API holes the docs writers found
- Anonymous callers can no longer create issues, comments or reactions on
public repositories; anonymous git fetches still work.
- Token scopes are enforced: a read-scoped token only reads, account-level
writes need write scope, and orgs, invitations, agents and tokens need
admin scope. Scopes are validated on the server.
- Agents can't add, change or delete secrets or variables at any level.
- A merge commit is refused on branches that require linear history, and
admins bypass unmet rules only with an explicit, audited bypass_rules.
- Drafts can be marked ready for review (PATCH draft, web button, CLI
`pr ready`).
- Failed webhook deliveries are retried with backoff; workflow_dispatch and
edited events are emitted.
- The exact branch protection rule wins over patterns, then the oldest.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AizKXSsengXZRegJxRATkL
* Account-level changes refuse tokens limited to repositories
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AizKXSsengXZRegJxRATkL
* Source archive cache under archives/; repository deletion removes Actions objects
The zipball/tarball cache (from the REST merge) lived under
repos/<id>/archives/, relying on the 90-day rule on repos/ that no longer
exists. It moves to archives/<id>/, deleted after 30 days (rebuilt on a miss).
Deleting a repository wiped its Durable Object but left its R2 objects for a
cleanup job that doesn't exist. Under repos/ they used to age out; caches
under actions-cache/ would now stay forever. destroy() deletes the
repository's objects under actions-cache/, artifacts/, logs/,
actions-staging/ and archives/, a page at a time within the subrequest
budget.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AizKXSsengXZRegJxRATkL
* JSON 404s for unknown API paths; self-hosted domains get their own route
Unknown /api paths returned the web UI's HTML 404; they now answer like
GitHub, in JSON. render-config only changed PUBLIC_URL for another domain,
leaving the route and email sender on diffinery.com; they now follow
DIFFINERY_DOMAIN (with DIFFINERY_ZONE and MAIL_DOMAIN to override).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AizKXSsengXZRegJxRATkL
* Event payloads report the repository's real visibility
Workflow event payloads always said repository.private was true, and webhook
payloads left private and visibility out. Both now carry GitHub's full
repository object from the directory, so a public repository reads as
private: false, visibility: public. release and repository_dispatch runs get
the same object and a sender.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AizKXSsengXZRegJxRATkL
* Cache scopes for older runners and merge groups; expired artifacts stay listed
Checked against GitHub's behaviour:
- Cache scoping. The cache service (Twirp v2, and the v1 artifactcache API
that actions/cache uses on hosts other than github.com) already saved to
the job's ref and restored from it, then the pull request's base, then the
default branch, restore-key prefixes included. The older runner's cache API
(/runner/jobs/:id/cache) had no scoping: any branch restored any other's
entries, and a blob could be fetched by key. Its entries are now stored as
`<ref>\n<key>`, looked up in the same order, and a blob is served only from
a scope the job may read. Merge group runs also read their target branch's
caches (merge_group.base_ref), as on GitHub.
- Artifact retention. retention-days already reached the server (the
built-in and upload-artifact v4 send expires_at) and was clamped to
1..90 days, but an expired artifact disappeared from the REST API. Now its
content is deleted, it is listed with `expired: true` for 90 more days,
its download gets 410, and it no longer counts against the quota. Expiry
also runs when a job ends, not only when uploads start.
Unit tests run the Runtime on a real SQLite schema (node:sqlite). E2E: the
built-in and the real actions/cache across branches and pull requests,
retention-days of the built-in and the real upload-artifact (and the 90-day
cap), and the older runner's cache API.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AizKXSsengXZRegJxRATkL
* Push protection covers the web editor, the commits API and MCP
RepoDO.commitFiles now runs the same secret scan as git push before it
writes anything, so the web editor, POST /diffinery/commits, the contents
API and MCP commit_files all refuse credential-shaped content. A human
repository admin can commit anyway through the commits API with an
allow_secrets reason; that is audited as api.secret_bypassed and emails
the repository's admins. Blocked API writes are audited against the
caller instead of a generic "api" actor.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AizKXSsengXZRegJxRATkL
* Sync each ref in its own queue so branch pushes stop cancelling main syncs
All sync runs shared one concurrency group, and GitHub keeps only the
newest pending run per group, so a push to any branch replaced a waiting
main sync and main's deploy went unverified. Each push event now copies
only its own ref in a queue of its own; a manual run still copies every
branch and tag.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AizKXSsengXZRegJxRATkL
* Archived repositories refuse every write; fixes from docs review batch 3
- Archived repositories refuse every write whatever the route's role
(issues, comments, reviews, protection, collaborators, secrets,
variables, hooks, MCP); unarchiving, deleting, watching and package
links still work.
- Renaming a repository or environment variable onto an existing name
is a 409 instead of overwriting it.
- A job's check_run_url resolves to the check run its job reports.
- Every workflow event payload carries repository and sender
(repository_dispatch, release, ...).
- Jobs with environment: emit a deployment event as well as
deployment_status.
- Deployment environments are case-insensitive, and a successful
status only makes older deployments inactive, not newer ones.
- An environment's html_url points at a page that exists.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AizKXSsengXZRegJxRATkL
* Run jobs as a needs graph, cascade skips and evaluate job keys per matrix leg
- RunWorkflow walks the job graph: a job starts as soon as its own needs
finish instead of waiting for its whole level. Steps per job are
unchanged, and runs planned before this replay their recorded levels.
- A job's if: sees all of its ancestors, like GitHub: a skipped, failed or
cancelled ancestor makes success() false, so skips cascade down the chain
and needs.<job>.result stays skipped. failure() sees any failed ancestor.
- runs-on, name, timeout-minutes, continue-on-error and job concurrency are
evaluated for each matrix leg (runs-on: ${{ matrix.os }} works, also as a
list); environment gets the needs and strategy contexts.
- Job-level continue-on-error now works: the failed leg doesn't fail the
run, doesn't trigger fail-fast and counts as a success for its dependents.
- Job-level concurrency groups: one job per group runs at a time across the
repository, others wait as pending, cancel-in-progress cancels them.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AizKXSsengXZRegJxRATkL
* E2E: job graph scheduling, skip cascades, per-leg runs-on, continue-on-error, job concurrency
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AizKXSsengXZRegJxRATkL
* Docs review batch 4: token minting, settings audit, merge bypass scopes, anonymous pushes
- A token limited to some repositories can no longer mint an unlimited
one; tokens can only be limited to repositories the account can see.
- A read token's write to a private repository it can't see is a 404,
not a scope error that reveals the repository.
- Settings changed through the API (repository, collaborators, branch
protection, secrets, variables, environment variables, webhooks, org
secrets) are audited like the web settings pages.
- bypass_merge_queue skips only the queue; bypass_rules skips unmet
rules. Bypassing linear history or the queue is audited too.
- The web merge form offers squash first and hides merge commits on
linear-history branches, and hides the bypass when enforce_admins is
on. Saving branch protection on the web keeps required check sources.
- The timeline event for converting to draft is convert_to_draft, as on
GitHub (the webhook action stays converted_to_draft).
- Labels and assignees from callers without triage access are dropped.
- An anonymous git push gets a 401 with WWW-Authenticate, so git offers
its stored credentials (public repositories included).
- deploy.yml follows DIFFINERY_DOMAIN, DIFFINERY_ZONE and MAIL_DOMAIN
and deploys from any Diffinery; the sync copies those variables.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AizKXSsengXZRegJxRATkL
* Manual syncs queue with main and lease each ref; production flag ignores case
A manual sync copies every ref, so it now shares main's queue and leases
each ref to the value Diffinery had just before it fetched from GitHub:
it can no longer put an older commit back over one another run copied.
Deployments to an existing "Production" environment named in another
case are still production.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AizKXSsengXZRegJxRATkL
* E2E: give each wrangler dev its own inspector port
The security suite's Worker failed to start on CI with "bind(): Address
already in use": every suite's wrangler dev used the default inspector
port, and the previous suite's runtime was still shutting down on it.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AizKXSsengXZRegJxRATkL
* Failed uploads give back their quota; older runners' artifacts respect what's left of it
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AizKXSsengXZRegJxRATkL
* E2E: retry a request once when wrangler dev closed the idle keep-alive connection
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AizKXSsengXZRegJxRATkL
---------
Co-authored-by: Claude <noreply@anthropic.com>Canberk Demirci authored · GitHub committed signed1 parent c9005decommit 4cfbb132936ef465a312dbf3e2c8fee4a8d18bebBrowse files
69 files changed+6916−635
Files (69)
- .github/workflows/deploy.yml+22−15
- .github/workflows/sync-to-diffinery.yml+42−13
- README.md+102−6
- apps/diffinery/src/app.ts+15−25
- apps/diffinery/src/auth/principal.ts+18−5
- apps/diffinery/src/docs/api-meta.ts+6−1
- apps/diffinery/src/env.ts+2−0
- apps/diffinery/src/http/actions-runtime.ts+638−0
- apps/diffinery/src/http/api.ts+47−41
- apps/diffinery/src/http/common.ts+25−3
- apps/diffinery/src/http/deployments.ts+8−2
- apps/diffinery/src/http/git.ts+4−1
- apps/diffinery/src/http/misc.ts+6−3
- apps/diffinery/src/http/runner.ts+37−12
- apps/diffinery/src/lib/blob-store.ts+391−0
- apps/diffinery/src/lib/job-permissions.ts+180−0
- apps/diffinery/src/lib/runtime-auth.ts+128−0
- apps/diffinery/src/lib/storage-keys.ts+104−0
- apps/diffinery/src/lib/util.ts+1−1
- apps/diffinery/src/lib/webhook-payload.ts+29−0
- apps/diffinery/src/mail/api.ts+2−2
- apps/diffinery/src/mail/notify.ts+1−0
- apps/diffinery/src/mail/web.tsx+2−2
- apps/diffinery/src/registry/api.ts+1−1
- apps/diffinery/src/repo/RepoDO.ts+141−6
- apps/diffinery/src/repo/actions.ts+297−104
- apps/diffinery/src/repo/collab.ts+26−7
- apps/diffinery/src/repo/deployments.ts+19−14
- apps/diffinery/src/repo/event-payloads.ts+226−0
- apps/diffinery/src/repo/extra-triggers.ts+4−1
- apps/diffinery/src/repo/gitdata.ts+9−7
- apps/diffinery/src/repo/job-graph.ts+104−0
- apps/diffinery/src/repo/runtime.ts+501−0
- apps/diffinery/src/repo/schema.ts+32−0
- apps/diffinery/src/repo/token-policy.ts+141−0
- apps/diffinery/src/repo/types.ts+2−0
- apps/diffinery/src/runner/workflow.ts+21−19
- apps/diffinery/src/web/collab.tsx+4−4
- apps/diffinery/src/web/packages.tsx+1−1
- apps/diffinery/src/web/repo-settings.tsx+9−3
- apps/diffinery/src/web/ui.tsx+1−1
- apps/diffinery/test/e2e/e2e.test.mjs+710−7
- apps/diffinery/test/e2e/email.test.mjs+1−1
- apps/diffinery/test/e2e/passkeys.test.mjs+1−1
- apps/diffinery/test/e2e/registry.test.mjs+3−1
- apps/diffinery/test/e2e/security.test.mjs+138−3
- apps/diffinery/test/unit/actions-runtime.test.ts+517−0
- apps/diffinery/test/unit/job-graph.test.ts+139−0
- apps/diffinery/test/unit/job-permissions.test.ts+161−0
- apps/diffinery/test/unit/webhook-payload.test.ts+36−0
- containers/runner/Dockerfile+14−1
- containers/runner/src/api.ts+0−32
- containers/runner/src/builtins/artifacts.ts+384−59
- containers/runner/src/builtins/cache.ts+131−68
- containers/runner/src/builtins/checkout.ts+70−27
- containers/runner/src/builtins/index.ts+63−17
- containers/runner/src/builtins/setup-go.ts+10−8
- containers/runner/src/builtins/setup-node.ts+59−29
- containers/runner/src/builtins/setup-python.ts+42−13
- containers/runner/src/builtins/tools.ts+24−15
- containers/runner/src/job.ts+28−3
- containers/runner/src/runtime.ts+151−0
- containers/runner/src/steps/actions.ts+57−3
- containers/runner/src/zip.ts+307−0
- containers/runner/test/helpers.ts+101−36
- containers/runner/test/job.test.ts+193−7
- containers/runner/test/runtime.test.ts+191−0
- packages/shared/src/runner.ts+14−0
- scripts/bootstrap.sh+22−4
modified.github/workflows/deploy.yml+22−15View file
| @@ -1,8 +1,12 @@ | ||
| 1 | 1 | name: Deploy |
| 2 | 2 | |
| 3 | -# Deploys diffinery.com from main. Uses the repository secrets | |
| 4 | -# CLOUDFLARE_API_TOKEN and CLOUDFLARE_ACCOUNT_ID; optional repository variable | |
| 5 | -# ACCESS_TEAM_DOMAIN (e.g. codician.cloudflareaccess.com). | |
| 3 | +# Deploys Diffinery from main. Uses the repository secrets | |
| 4 | +# CLOUDFLARE_API_TOKEN and CLOUDFLARE_ACCOUNT_ID, and these optional repository | |
| 5 | +# variables (the same ones scripts/bootstrap.sh reads): | |
| 6 | +# DIFFINERY_DOMAIN where it is served (default diffinery.com) | |
| 7 | +# DIFFINERY_ZONE the Cloudflare zone for the route (default: the domain's last two labels) | |
| 8 | +# MAIL_DOMAIN sends from notifications@<MAIL_DOMAIN> (default DIFFINERY_DOMAIN) | |
| 9 | +# ACCESS_TEAM_DOMAIN e.g. codician.cloudflareaccess.com | |
| 6 | 10 | on: |
| 7 | 11 | push: |
| 8 | 12 | branches: [main] |
| @@ -18,16 +22,19 @@ | ||
| 18 | 22 | jobs: |
| 19 | 23 | deploy: |
| 20 | 24 | name: Deploy to Cloudflare |
| 21 | - # Production deploys from Diffinery, on its own hosted runners. GitHub | |
| 22 | - # mirrors this repository there (see sync-to-diffinery.yml); running this | |
| 25 | + # Production deploys from Diffinery itself, on its own hosted runners (any | |
| 26 | + # Diffinery, so self-hosted copies deploy themselves too). Running this | |
| 23 | 27 | # workflow by hand on GitHub still deploys, as a fallback. |
| 24 | - if: ${{ github.server_url == 'https://diffinery.com' || github.event_name == 'workflow_dispatch' }} | |
| 28 | + if: ${{ github.server_url != 'https://github.com' || github.event_name == 'workflow_dispatch' }} | |
| 25 | 29 | runs-on: ubuntu-latest |
| 26 | 30 | timeout-minutes: 45 |
| 27 | 31 | environment: |
| 28 | 32 | name: production |
| 29 | - url: https://diffinery.com | |
| 33 | + url: https://${{ vars.DIFFINERY_DOMAIN || 'diffinery.com' }} | |
| 30 | 34 | env: |
| 35 | + DIFFINERY_DOMAIN: ${{ vars.DIFFINERY_DOMAIN || 'diffinery.com' }} | |
| 36 | + DIFFINERY_ZONE: ${{ vars.DIFFINERY_ZONE }} | |
| 37 | + MAIL_DOMAIN: ${{ vars.MAIL_DOMAIN }} | |
| 31 | 38 | CLOUDFLARE_API_TOKEN: ${{ secrets.CLOUDFLARE_API_TOKEN }} |
| 32 | 39 | CLOUDFLARE_ACCOUNT_ID: ${{ secrets.CLOUDFLARE_ACCOUNT_ID }} |
| 33 | 40 | ACCESS_TEAM_DOMAIN: ${{ vars.ACCESS_TEAM_DOMAIN }} |
| @@ -65,24 +72,24 @@ | ||
| 65 | 72 | run: | |
| 66 | 73 | # A new custom domain can take a few minutes to get its certificate. |
| 67 | 74 | for i in $(seq 1 30); do |
| 68 | - code=$(curl -s -o /dev/null -w '%{http_code}' https://diffinery.com/healthz) && [ "$code" = 200 ] && exit 0 | |
| 75 | + code=$(curl -s -o /dev/null -w '%{http_code}' https://$DIFFINERY_DOMAIN/healthz) && [ "$code" = 200 ] && exit 0 | |
| 69 | 76 | sleep 10 |
| 70 | 77 | done |
| 71 | - echo "::error::https://diffinery.com/healthz did not return 200 (last: $code). Check that the diffinery.com DNS record is proxied (orange cloud)."; exit 1 | |
| 78 | + echo "::error::https://$DIFFINERY_DOMAIN/healthz did not return 200 (last: $code). Check that the $DIFFINERY_DOMAIN DNS record is proxied (orange cloud)."; exit 1 | |
| 72 | 79 | |
| 73 | 80 | - name: Check sign-in pages |
| 74 | 81 | run: | |
| 75 | 82 | set -u |
| 76 | - page=$(curl -sS https://diffinery.com/login) | |
| 83 | + page=$(curl -sS https://$DIFFINERY_DOMAIN/login) | |
| 77 | 84 | grep -q "Sign in with a passkey" <<<"$page" || { echo "::error::/login is not the Diffinery sign-in page"; exit 1; } |
| 78 | 85 | echo "/login serves the passkey sign-in page" |
| 79 | - loc=$(curl -sS -o /dev/null -w '%{http_code} %{redirect_url}' https://diffinery.com/auth/access) | |
| 86 | + loc=$(curl -sS -o /dev/null -w '%{http_code} %{redirect_url}' https://$DIFFINERY_DOMAIN/auth/access) | |
| 80 | 87 | echo "/auth/access -> $loc" |
| 81 | 88 | [[ "$loc" == 302\ https://*.cloudflareaccess.com/* ]] || echo "::warning::/auth/access is not behind Cloudflare Access yet, so email recovery and first admin sign-in will not work" |
| 82 | 89 | for kind in js css; do |
| 83 | 90 | asset=$(grep -o "/-/kumo/[a-z0-9]*\.$kind" <<<"$page" | head -1) |
| 84 | 91 | [ -n "$asset" ] || { echo "::error::/login does not load the Kumo $kind bundle"; exit 1; } |
| 85 | - code=$(curl -sS -o /dev/null -w '%{http_code}' "https://diffinery.com$asset") | |
| 92 | + code=$(curl -sS -o /dev/null -w '%{http_code}' "https://$DIFFINERY_DOMAIN$asset") | |
| 86 | 93 | [ "$code" = 200 ] || { echo "::error::$asset returned $code"; exit 1; } |
| 87 | 94 | echo "$asset -> 200" |
| 88 | 95 | done |
| @@ -90,15 +97,15 @@ | ||
| 90 | 97 | run: | |
| 91 | 98 | set -u |
| 92 | 99 | for p in /favicon.ico /favicon.svg /favicon-48.png /apple-touch-icon.png /og-image.png /site.webmanifest /robots.txt /sitemap.xml; do |
| 93 | - code=$(curl -sS -o /dev/null -w '%{http_code} %{content_type}' "https://diffinery.com$p") | |
| 100 | + code=$(curl -sS -o /dev/null -w '%{http_code} %{content_type}' "https://$DIFFINERY_DOMAIN$p") | |
| 94 | 101 | [[ "$code" == 200\ * ]] || { echo "::error::$p returned $code"; exit 1; } |
| 95 | 102 | echo "$p -> $code" |
| 96 | 103 | done |
| 97 | - home=$(curl -sS https://diffinery.com/) | |
| 104 | + home=$(curl -sS https://$DIFFINERY_DOMAIN/) | |
| 98 | 105 | for tag in 'rel="icon" href="/favicon.ico"' 'property="og:image"' 'rel="canonical"' 'application/ld+json'; do |
| 99 | 106 | grep -qF "$tag" <<<"$home" || { echo "::error::the home page is missing $tag"; exit 1; } |
| 100 | 107 | done |
| 101 | 108 | echo "the home page carries the favicon, canonical, Open Graph and structured data tags" |
| 102 | - bm=$(curl -sS -D - -o /dev/null https://diffinery.com/sitemap.xml | tr -d '\r' | sed -n 's/^x-d1-bookmark: //Ip') | |
| 109 | + bm=$(curl -sS -D - -o /dev/null https://$DIFFINERY_DOMAIN/sitemap.xml | tr -d '\r' | sed -n 's/^x-d1-bookmark: //Ip') | |
| 103 | 110 | [ -n "$bm" ] || { echo "::error::D1 queries don't run in a session (no x-d1-bookmark header)"; exit 1; } |
| 104 | 111 | echo "D1 sessions are on (bookmark $bm)" |
modified.github/workflows/sync-to-diffinery.yml+42−13View file
| @@ -19,15 +19,21 @@ | ||
| 19 | 19 | permissions: |
| 20 | 20 | contents: read |
| 21 | 21 | |
| 22 | +# One queue per ref: a push to some other branch never replaces a waiting main sync. | |
| 23 | +# Each push event copies only its own ref, so runs for different refs can't undo | |
| 24 | +# each other. A manual run copies every branch and tag; it queues with main, and | |
| 25 | +# each ref it copies is leased to what Diffinery had just before, so it can't put | |
| 26 | +# back an older commit over one another run copied meanwhile. | |
| 22 | 27 | concurrency: |
| 23 | - group: sync-to-diffinery | |
| 28 | + group: sync-to-diffinery-${{ github.event_name == 'push' && github.ref || 'refs/heads/main' }} | |
| 24 | 29 | cancel-in-progress: false |
| 25 | 30 | |
| 26 | 31 | jobs: |
| 27 | 32 | move: |
| 28 | 33 | name: Copy code, secrets and variables, then check Actions on Diffinery |
| 29 | 34 | # Runs on GitHub only; the copy on Diffinery has nothing to sync to. |
| 30 | - if: ${{ github.server_url == 'https://github.com' }} | |
| 35 | + # Branch deletions aren't copied (Diffinery keeps the branch until it's deleted there). | |
| 36 | + if: ${{ github.server_url == 'https://github.com' && !github.event.deleted }} | |
| 31 | 37 | runs-on: ubuntu-latest |
| 32 | 38 | timeout-minutes: 90 |
| 33 | 39 | environment: production |
| @@ -65,6 +71,9 @@ | ||
| 65 | 71 | S_CLOUDFLARE_ACCOUNT_ID: ${{ secrets.CLOUDFLARE_ACCOUNT_ID }} |
| 66 | 72 | S_DIFFINERY_LIVE_TOKEN: ${{ secrets.DIFFINERY_LIVE_TOKEN }} |
| 67 | 73 | V_ACCESS_TEAM_DOMAIN: ${{ vars.ACCESS_TEAM_DOMAIN }} |
| 74 | + V_DIFFINERY_DOMAIN: ${{ vars.DIFFINERY_DOMAIN }} | |
| 75 | + V_DIFFINERY_ZONE: ${{ vars.DIFFINERY_ZONE }} | |
| 76 | + V_MAIL_DOMAIN: ${{ vars.MAIL_DOMAIN }} | |
| 68 | 77 | run: | |
| 69 | 78 | set -euo pipefail |
| 70 | 79 | put() { curl -fsS -o /dev/null -X PUT "$DIFFINERY/api/v3/repos/$ORG/$NAME/actions/$1/$2" -H "Authorization: Bearer $TOKEN" -H "content-type: application/json" --data-binary @-; } |
| @@ -72,7 +81,7 @@ | ||
| 72 | 81 | v="S_$name" |
| 73 | 82 | if [ -n "${!v}" ]; then jq -n --arg v "${!v}" '{value: $v}' | put secrets "$name"; echo "secret $name copied"; else echo "::warning::secret $name is empty here, so it was not copied"; fi |
| 74 | 83 | done |
| 75 | - for name in ACCESS_TEAM_DOMAIN; do | |
| 84 | + for name in ACCESS_TEAM_DOMAIN DIFFINERY_DOMAIN DIFFINERY_ZONE MAIL_DOMAIN; do | |
| 76 | 85 | v="V_$name" |
| 77 | 86 | if [ -n "${!v}" ]; then jq -n --arg v "${!v}" '{value: $v}' | put variables "$name"; echo "variable $name = ${!v}"; fi |
| 78 | 87 | done |
| @@ -82,24 +91,44 @@ | ||
| 82 | 91 | fetch-depth: 0 |
| 83 | 92 | lfs: true |
| 84 | 93 | |
| 85 | - - name: Push every branch and tag | |
| 94 | + - name: Push the branch or tag (a manual run pushes all of them) | |
| 86 | 95 | env: |
| 87 | 96 | ALLOW_SECRETS: ${{ inputs.allow_secrets }} |
| 88 | 97 | run: | |
| 89 | 98 | set -euo pipefail |
| 90 | - git fetch -q --tags origin '+refs/heads/*:refs/remotes/origin/*' | |
| 91 | 99 | remote="https://x-token:${TOKEN}@${DIFFINERY#https://}/$ORG/$NAME.git" |
| 92 | - refs=$(git for-each-ref --format='+%(refname):refs/heads/%(refname:lstrip=3)' refs/remotes/origin | grep -v ':refs/heads/HEAD$') | |
| 93 | - git push -q ${ALLOW_SECRETS:+-o "allow-secrets=$ALLOW_SECRETS"} "$remote" $refs '+refs/tags/*:refs/tags/*' | |
| 100 | + opts=(-q ${ALLOW_SECRETS:+-o "allow-secrets=$ALLOW_SECRETS"}) | |
| 101 | + if [ "$GITHUB_EVENT_NAME" = push ]; then | |
| 102 | + git push "${opts[@]}" "$remote" "+$GITHUB_SHA:$GITHUB_REF" | |
| 103 | + want=$GITHUB_SHA | |
| 104 | + else | |
| 105 | + # Read Diffinery's refs first, then GitHub's: GitHub's are then at least as new. | |
| 106 | + declare -A there | |
| 107 | + while read -r sha ref; do there[$ref]=$sha; done < <(git ls-remote "$remote" 'refs/heads/*' 'refs/tags/*') | |
| 108 | + git fetch -q --tags --force origin '+refs/heads/*:refs/remotes/origin/*' | |
| 109 | + args=() | |
| 110 | + while read -r ref; do | |
| 111 | + dest="refs/heads/${ref#refs/remotes/origin/}" | |
| 112 | + args+=("--force-with-lease=$dest:${there[$dest]:-}" "$ref:$dest") | |
| 113 | + done < <(git for-each-ref --format='%(refname)' refs/remotes/origin | grep -v '/HEAD$') | |
| 114 | + for tag in $(git for-each-ref --format='%(refname)' refs/tags); do | |
| 115 | + args+=("--force-with-lease=$tag:${there[$tag]:-}" "$tag:$tag") | |
| 116 | + done | |
| 117 | + git push "${opts[@]}" "$remote" "${args[@]}" | |
| 118 | + want=$(git rev-parse refs/remotes/origin/main) | |
| 119 | + fi | |
| 94 | 120 | if git lfs ls-files | grep -q .; then git lfs push --all "$remote"; fi |
| 95 | 121 | echo "branches on Diffinery:" |
| 96 | - curl -fsS "$DIFFINERY/api/v3/repos/$ORG/$NAME/branches" -H "Authorization: Bearer $TOKEN" | jq -r '.[] | " \(.name) \(.commit.sha)"' | |
| 97 | - want=$(git rev-parse refs/remotes/origin/main) | |
| 98 | - got=$(curl -fsS "$DIFFINERY/api/v3/repos/$ORG/$NAME/branches" -H "Authorization: Bearer $TOKEN" | jq -r '.[] | select(.name=="main") | .commit.sha') | |
| 99 | - [ "$want" = "$got" ] || { echo "::error::main on Diffinery is $got, expected $want"; exit 1; } | |
| 100 | - echo "main matches GitHub at $want" | |
| 122 | + curl -fsS "$DIFFINERY/api/v3/repos/$ORG/$NAME/branches?per_page=100" -H "Authorization: Bearer $TOKEN" | jq -r '.[] | " \(.name) \(.commit.sha)"' | |
| 123 | + if [ "$GITHUB_REF" = refs/heads/main ]; then | |
| 124 | + got=$(curl -fsS "$DIFFINERY/api/v3/repos/$ORG/$NAME/branches/main" -H "Authorization: Bearer $TOKEN" | jq -r '.commit.sha') | |
| 125 | + [ "$want" = "$got" ] || { echo "::error::main on Diffinery is $got, expected $want"; exit 1; } | |
| 126 | + echo "main matches GitHub at $want" | |
| 127 | + fi | |
| 128 | + echo "want=$want" >> "$GITHUB_OUTPUT" | |
| 129 | + id: push | |
| 101 | 130 | |
| 102 | 131 | - name: Wait for Actions on Diffinery |
| 103 | 132 | # Pushes to main start CI (and, once Diffinery deploys, the deploy) there. |
| 104 | 133 | if: github.ref == 'refs/heads/main' |
| 105 | - run: scripts/diffinery-wait-runs.sh "$ORG/$NAME" "$(git rev-parse refs/remotes/origin/main)" 75 | |
| 134 | + run: scripts/diffinery-wait-runs.sh "$ORG/$NAME" "${{ steps.push.outputs.want }}" 75 |
modifiedREADME.md+102−6View file
| @@ -81,6 +81,29 @@ | ||
| 81 | 81 | generates `MASTER_KEY` and the OIDC signing key inside the job, so neither |
| 82 | 82 | value is ever printed. |
| 83 | 83 | |
| 84 | +### Storage | |
| 85 | + | |
| 86 | +Durable and transient data live under separate R2 prefixes, because lifecycle | |
| 87 | +rules can only select by prefix (`apps/diffinery/src/lib/storage-keys.ts`): | |
| 88 | + | |
| 89 | +| Prefix | What | Lifecycle rule | | |
| 90 | +|---|---|---| | |
| 91 | +| `repos/` | git objects, LFS objects, release assets | none: kept until the repository is deleted | | |
| 92 | +| `registry/` | container images | none: unreferenced blobs are swept by the cron job | | |
| 93 | +| `actions-cache/` | Actions caches | none: Diffinery evicts unused entries after 7 days, and the least recently used above 10 GB per repository | | |
| 94 | +| `artifacts/` | workflow artifacts | deleted after 90 days (shorter `retention-days` are handled by Diffinery) | | |
| 95 | +| `logs/` | archived job logs | deleted after 90 days, like GitHub's log retention | | |
| 96 | +| `actions-staging/` | uploads in progress | deleted after a day | | |
| 97 | +| `archives/` | zipball and tarball cache | deleted after 30 days (rebuilt when asked for again) | | |
| 98 | +| `events/` | the event archive | deleted after 400 days | | |
| 99 | + | |
| 100 | +Unfinished multipart uploads are aborted after a day everywhere. `bootstrap.sh` | |
| 101 | +applies these rules on every deploy and stops the deploy if that fails, or if | |
| 102 | +any rule would still delete objects under `repos/`. Artifacts, caches and logs | |
| 103 | +written before this layout (under `repos/<id>/...`) are still read from there. | |
| 104 | +Deleting a repository deletes its objects under the Actions and archive | |
| 105 | +prefixes. | |
| 106 | + | |
| 84 | 107 | If the token can't read your Zero Trust organization, set a repository |
| 85 | 108 | **variable** `ACCESS_TEAM_DOMAIN` (for example `codician.cloudflareaccess.com`). |
| 86 | 109 | |
| @@ -237,12 +260,47 @@ | ||
| 237 | 260 | |
| 238 | 261 | - `needs` and job outputs, `if` with `success()`, `failure()`, `always()` and |
| 239 | 262 | `cancelled()`, `strategy.matrix` with include and exclude, `fail-fast` and |
| 240 | - `max-parallel`, `concurrency` with `cancel-in-progress`, and `timeout-minutes`. | |
| 263 | + `max-parallel`, `concurrency` with `cancel-in-progress` (for runs and for | |
| 264 | + jobs), and `timeout-minutes`. | |
| 265 | +- A job starts as soon as the jobs it `needs` have finished. It doesn't wait | |
| 266 | + for unrelated jobs. | |
| 267 | +- As on GitHub, a job is skipped when a job above it in the `needs` chain | |
| 268 | + failed, was cancelled or was skipped, unless its `if:` uses `always()`, | |
| 269 | + `failure()` or `cancelled()`. `!cancelled() && !failure()` runs it after a | |
| 270 | + skip. `needs.<job>.result` is `skipped` for a skipped job. | |
| 271 | +- `runs-on`, `name`, `environment`, `timeout-minutes`, `continue-on-error` | |
| 272 | + and job `concurrency` can use the `matrix` and `strategy` contexts, and are | |
| 273 | + worked out for each matrix job, e.g. `runs-on: ${{ matrix.os }}`. | |
| 274 | +- A job-level `concurrency` group lets one job run at a time across the | |
| 275 | + repository. Others wait as pending; a newer pending job replaces an older | |
| 276 | + one. | |
| 277 | +- Job `continue-on-error`: a failed job doesn't fail the run, doesn't cancel | |
| 278 | + the rest of its matrix, and counts as a success for the jobs that need it. | |
| 241 | 279 | - `continue-on-error`, `defaults.run`, `env` at every level, `$GITHUB_OUTPUT`, |
| 242 | 280 | `$GITHUB_ENV`, `$GITHUB_PATH`, `$GITHUB_STEP_SUMMARY`, and masking of |
| 243 | 281 | secrets and `::add-mask::` values. |
| 244 | -- `secrets.*`, `vars.*`, `GITHUB_TOKEN` (a per-job token), and | |
| 245 | - `permissions: id-token: write` for OIDC (issuer `https://diffinery.com`). | |
| 282 | +- `secrets.*`, `vars.*`, and `GITHUB_TOKEN`, a per-job token that follows | |
| 283 | + GitHub's `permissions:` model: workflow- or job-level `read-all`, | |
| 284 | + `write-all`, `{}` or per-scope `contents`, `issues`, `pull-requests`, | |
| 285 | + `statuses`, `checks`, `deployments`, `actions`, `packages`, `pages` and so | |
| 286 | + on. Without `permissions:` a job gets the repository's default, which is | |
| 287 | + GitHub's restricted one (`contents` and `packages` read) for new | |
| 288 | + repositories, and write for ones created before the setting existed. An | |
| 289 | + admin changes it, and whether | |
| 290 | + workflows may create and approve pull requests (off by default), with | |
| 291 | + `PUT /repos/{owner}/{repo}/actions/permissions/workflow` | |
| 292 | + (`default_workflow_permissions`, `can_approve_pull_request_reviews`). Even | |
| 293 | + when allowed, a workflow's approval doesn't count toward required approvals. | |
| 294 | + A reusable workflow never gets more than its caller, and a pull request from | |
| 295 | + another repository only reads. Each API endpoint needs its scope (an issue | |
| 296 | + write without `issues: write` is a 403, like GitHub's "Resource not | |
| 297 | + accessible by integration"). The token never administers the repository, | |
| 298 | + reads secrets or creates or changes workflow files (GitHub's `workflows` | |
| 299 | + permission), and branch protection applies to it. A run whose code came from | |
| 300 | + an agent (the agent pushed it or the branch, or opened the pull request, and | |
| 301 | + re-runs of such runs) is held to the agent rules: its token can't push or | |
| 302 | + merge to the default or protected branches. `permissions: id-token: write` | |
| 303 | + enables OIDC (issuer `https://diffinery.com`). | |
| 246 | 304 | - `runs-on: ubuntu-latest` (and any `ubuntu-*` or `linux` label) runs in a |
| 247 | 305 | Cloudflare Container. `runs-on: [self-hosted, …]` jobs wait for a runner you |
| 248 | 306 | start with `diffinery runner -R owner/repo --labels gpu,macos`. Use that for |
| @@ -266,10 +324,48 @@ | ||
| 266 | 324 | token, then push and pull `diffinery.com/<owner>/<name>`. Images are |
| 267 | 325 | private by default and can be linked to a repository. See |
| 268 | 326 | [Container registry](https://diffinery.com/docs/packages/container-registry). |
| 327 | +- Actions from GitHub run unmodified, including `actions/github-script`, | |
| 328 | + `setup-java`, `setup-dotnet` and the rest. Jobs get the same runtime | |
| 329 | + services GitHub provides (`ACTIONS_RUNTIME_TOKEN`, `ACTIONS_RESULTS_URL`, | |
| 330 | + `ACTIONS_CACHE_URL`), scoped to the job and expiring with it, so | |
| 331 | + `actions/cache` (v3 and v4), `actions/upload-artifact` and | |
| 332 | + `actions/download-artifact` (v4: `pattern`, `merge-multiple`, `run-id` with | |
| 333 | + `github-token`) and `docker/build-push-action` with `cache-to: type=gha` store | |
| 334 | + caches and artifacts in Diffinery. `GITHUB_EVENT_PATH` holds a complete event | |
| 335 | + payload and `github.run_id` is numeric. | |
| 336 | + Caches and artifacts are listed and deleted through the REST API | |
| 337 | + (`/actions/caches`, `/actions/artifacts`). Each repository keeps up to 10 GB | |
| 338 | + of caches (the least recently used are evicted) and 10 GB of artifacts | |
| 339 | + (`ARTIFACT_QUOTA_BYTES` changes it; uploads beyond it are refused). Uploads | |
| 340 | + in progress count toward both, and a job's unfinished uploads are removed | |
| 341 | + when it ends. As on GitHub, a cache is saved for the ref the job runs on and | |
| 342 | + restored from that ref, then the pull request's base branch, then the | |
| 343 | + default branch, so branches and pull requests never see each other's | |
| 344 | + caches. Artifacts keep their `retention-days` (1 to 90 days, 90 by default); | |
| 345 | + an expired artifact's content is deleted, and the REST API still lists it | |
| 346 | + with `expired: true` (downloads get 410) for another 90 days. | |
| 347 | +- `actions/checkout`, `cache`, `upload-artifact`, `download-artifact`, | |
| 348 | + `setup-node`, `setup-python` and `setup-go` run as built-ins for speed. They | |
| 349 | + share storage and formats with the real actions (a cache saved by one | |
| 350 | + restores with the other) and support their inputs, including checkout's | |
| 351 | + `lfs`, `submodules`, `sparse-checkout`, `filter`, `fetch-tags`, `ref` and | |
| 352 | + `persist-credentials: false`, and the `cache:` input of the setup actions. | |
| 353 | + A built-in hands over to the real action when it sees an input or ref it | |
| 354 | + doesn't mirror (a commit SHA, a branch, a newer major). To always run the | |
| 355 | + real action, set `DIFFINERY_BUILTINS: false` in a workflow, job or step | |
| 356 | + `env:`, or start a self-hosted runner with `DIFFINERY_DISABLE_BUILTINS=1`. | |
| 357 | +- Hosted runners look like GitHub's `ubuntu-24.04` image to actions: | |
| 358 | + `ImageOS=ubuntu24`, `RUNNER_TOOL_CACHE=/opt/hostedtoolcache`, and `git`, | |
| 359 | + `git-lfs`, `jq`, `zip`, `zstd`, `curl`, `python3`/`pip`, `build-essential`, | |
| 360 | + `docker` and `gh` installed. `ACTIONS_STEP_DEBUG` and `ACTIONS_RUNNER_DEBUG` | |
| 361 | + (as secrets or variables) turn on debug logging. | |
| 269 | 362 | |
| 270 | -**Not supported yet:** `actions/github-script`, | |
| 271 | -Windows and macOS hosted runners, and | |
| 272 | -`deepen-not` / partial clone filters. A job that uses an unsupported feature fails | |
| 363 | +**Not supported yet:** Windows and macOS hosted runners, the GraphQL API | |
| 364 | +(`github.graphql` in github-script), and `deepen-not` / partial clone filters | |
| 365 | +on the server (checkout's `filter` falls back to a full fetch). One cache or | |
| 366 | +artifact upload request goes through the Worker, so it is limited to the | |
| 367 | +Worker's request size (100 MB on most plans); the built-ins and the official | |
| 368 | +actions upload large files in blocks. A job that uses an unsupported feature fails | |
| 273 | 369 | with a clear message instead of misbehaving. |
| 274 | 370 | |
| 275 | 371 | ## REST API |
modifiedapps/diffinery/src/app.ts+15−25View file
| @@ -5,6 +5,7 @@ | ||
| 5 | 5 | import type { Env } from "./env.js"; |
| 6 | 6 | import { credentialFrom, resolvePrincipal, type AppVars } from "./auth/principal.js"; |
| 7 | 7 | import { Directory } from "./lib/directory.js"; |
| 8 | +import { getFirst } from "./lib/storage-keys.js"; | |
| 8 | 9 | import { BOOKMARK_COOKIE, BOOKMARK_HEADER, d1Session, sessionConstraint } from "./lib/d1.js"; |
| 9 | 10 | import { SESSION_COOKIE } from "./auth/sessions.js"; |
| 10 | 11 | import { newId } from "./lib/util.js"; |
| @@ -13,11 +14,12 @@ | ||
| 13 | 14 | import { apiNotFound, errorResponse, loadRepo, repoStub } from "./http/common.js"; |
| 14 | 15 | import { git } from "./http/git.js"; |
| 15 | 16 | import { runner } from "./http/runner.js"; |
| 17 | +import { actionsRuntime } from "./http/actions-runtime.js"; | |
| 16 | 18 | import { api, wellKnown } from "./http/api.js"; |
| 17 | 19 | import { avatars } from "./http/avatars.js"; |
| 18 | 20 | import { mailApi } from "./mail/api.js"; |
| 19 | 21 | import { files } from "./http/files.js"; |
| 20 | -import { nodeId, numericId } from "./http/gh.js"; | |
| 22 | +import { webhookPayload } from "./lib/webhook-payload.js"; | |
| 21 | 23 | import { mcp } from "./mcp/server.js"; |
| 22 | 24 | import { registry } from "./registry/http.js"; |
| 23 | 25 | import { packagesApi } from "./registry/api.js"; |
| @@ -50,9 +52,11 @@ | ||
| 50 | 52 | c.env = d1.env; |
| 51 | 53 | const dir = new Directory(c.env); |
| 52 | 54 | c.set("dir", dir); |
| 55 | + // The Actions runtime services authenticate with their own signed tokens and URLs (http/actions-runtime.ts). | |
| 56 | + const runtime = c.req.path.startsWith("/twirp/") || c.req.path.startsWith("/_runtime/"); | |
| 53 | 57 | // Avatar images are public and on every page many times over: skip the session lookup for them. |
| 54 | 58 | const avatar = (c.req.method === "GET" || c.req.method === "HEAD") && c.req.path.startsWith("/api/avatars/"); |
| 55 | - c.set("principal", avatar ? { type: "anonymous" } : await resolvePrincipal(c.req.raw, c.env, dir)); | |
| 59 | + c.set("principal", runtime || avatar ? { type: "anonymous" } : await resolvePrincipal(c.req.raw, c.env, dir)); | |
| 56 | 60 | await next(); |
| 57 | 61 | if (c.res.status === 101) return; |
| 58 | 62 | const h = c.res.headers; |
| @@ -79,6 +83,7 @@ | ||
| 79 | 83 | app.route("/", mailApi); |
| 80 | 84 | app.route("/", registry); |
| 81 | 85 | app.route("/", packagesApi); |
| 86 | +app.route("/", actionsRuntime); | |
| 82 | 87 | app.route("/", api); |
| 83 | 88 | app.route("/", files); |
| 84 | 89 | app.route("/", mcp); |
| @@ -92,15 +97,18 @@ | ||
| 92 | 97 | |
| 93 | 98 | app.get("/:owner/:repo/actions/runs/:runId/artifacts/:name", async (c) => { |
| 94 | 99 | const { stub } = await loadRepo(c, c.req.param("owner"), c.req.param("repo")); |
| 100 | + // By name, or by id (the artifact-url output of upload-artifact). | |
| 95 | 101 | const name = c.req.param("name"); |
| 96 | - if (!(await stub.artifacts(c.req.param("runId"))).some((a) => a.name === name)) return c.json({ message: "artifact not found" }, 404); | |
| 97 | - const obj = await c.env.BUCKET.get(await stub.artifactKeyForRun(c.req.param("runId"), name)); | |
| 102 | + const all = await stub.artifacts(c.req.param("runId")); | |
| 103 | + const artifact = all.find((a) => a.name === name) ?? (/^\d+$/.test(name) ? all.find((a) => a.id === Number(name)) : undefined); | |
| 104 | + if (!artifact) return c.json({ message: "artifact not found" }, 404); | |
| 105 | + const obj = await getFirst(c.env.BUCKET, artifact.r2Keys); | |
| 98 | 106 | if (!obj) return c.json({ message: "artifact expired" }, 410); |
| 99 | 107 | return new Response(obj.body, { |
| 100 | 108 | headers: { |
| 101 | - "content-type": "application/gzip", | |
| 109 | + "content-type": artifact.contentType, | |
| 102 | 110 | "content-length": String(obj.size), |
| 103 | - "content-disposition": `attachment; filename="${name.replace(/[^\w.-]+/g, "_")}.tar.gz"`, | |
| 111 | + "content-disposition": `attachment; filename="${artifact.fileName.replace(/[^\w.-]+/g, "_")}"`, | |
| 104 | 112 | }, |
| 105 | 113 | }); |
| 106 | 114 | }); |
| @@ -123,29 +131,11 @@ | ||
| 123 | 131 | audit: "audit", |
| 124 | 132 | }; |
| 125 | 133 | |
| 126 | -function webhookPayload(env: Env, ev: Exclude<DiffineryEvent, { type: "audit" }>) { | |
| 127 | - const repository = { id: numericId(ev.repoId), node_id: nodeId("Repository", ev.repoId), name: ev.repo, full_name: `${ev.owner}/${ev.repo}`, owner: { login: ev.owner }, html_url: `${env.PUBLIC_URL}/${ev.owner}/${ev.repo}`, clone_url: `${env.PUBLIC_URL}/${ev.owner}/${ev.repo}.git` }; | |
| 128 | - const sender = { id: ev.actor.id, login: ev.actor.login, type: ev.actor.kind === "agent" ? "Bot" : ev.actor.kind === "system" ? "Bot" : "User", on_behalf_of: ev.actor.onBehalfOf }; | |
| 129 | - const { type: _t, repoId: _r, owner: _o, repo: _n, actor: _a, ...rest } = ev; | |
| 130 | - const action = "action" in ev ? (ev.action === "merged" ? "closed" : ev.action) : undefined; | |
| 131 | - const extra: Record<string, unknown> = {}; | |
| 132 | - if (ev.type === "pull_request") { | |
| 133 | - extra.pull_request = { number: ev.number, html_url: `${env.PUBLIC_URL}/${ev.owner}/${ev.repo}/pull/${ev.number}`, head: { ref: ev.headRef, sha: ev.headSha }, base: { ref: ev.baseRef }, merged: ev.action === "merged" }; | |
| 134 | - } | |
| 135 | - if (ev.type === "issues" || ev.type === "issue_comment") extra.issue = { number: ev.number, html_url: `${env.PUBLIC_URL}/${ev.owner}/${ev.repo}/issues/${ev.number}` }; | |
| 136 | - // GitHub-shaped events (release, status, deployment, ...) carry their payload ready-made. | |
| 137 | - if ("payload" in ev) { | |
| 138 | - const { payload, at: _at, ...head } = rest as typeof rest & { payload: Record<string, unknown>; at: string }; | |
| 139 | - return { ...head, ...payload, action, repository, sender }; | |
| 140 | - } | |
| 141 | - return { ...rest, action, repository, sender, ...extra }; | |
| 142 | -} | |
| 143 | - | |
| 144 | 134 | async function deliverWebhooks(env: Env, ev: Exclude<DiffineryEvent, { type: "audit" }>): Promise<void> { |
| 145 | 135 | const stub = repoStub(env, ev.repoId); |
| 146 | 136 | const hooks = (await stub.webhooks()).filter((h) => h.active && (h.events.includes("*") || h.events.includes(GITHUB_EVENT[ev.type]))); |
| 147 | 137 | if (!hooks.length) return; |
| 148 | - const body = JSON.stringify(webhookPayload(env, ev)); | |
| 138 | + const body = JSON.stringify(webhookPayload(env.PUBLIC_URL, ev, await new Directory(env).repoById(ev.repoId))); | |
| 149 | 139 | // Each hook retries on its own (deliverWebhook requeues it). Once a hook may have been sent to, |
| 150 | 140 | // an error must not fail the stage, or the stage retry would send every hook again. |
| 151 | 141 | const sent = await Promise.allSettled(hooks.map((h) => deliverWebhook(webhookDeps(env), { type: "webhook_retry", repoId: ev.repoId, hookId: h.id, event: GITHUB_EVENT[ev.type], diffineryEvent: ev.type, body, delivery: newId(), attempt: 0 }, h))); |
modifiedapps/diffinery/src/auth/principal.ts+18−5View file
| @@ -22,13 +22,21 @@ | ||
| 22 | 22 | |
| 23 | 23 | /** Extract a bearer/basic credential, as git, curl, the CLI and MCP clients send it. */ |
| 24 | 24 | export function credentialFrom(req: Request): string | null { |
| 25 | - const h = req.headers.get("authorization"); | |
| 25 | + // A request can carry the header twice (git-lfs sends http.extraheader and the batch API's | |
| 26 | + // header), which arrives joined with ", ": use the first. | |
| 27 | + const h = req.headers.get("authorization")?.split(",")[0]!.trim(); | |
| 26 | 28 | if (!h) return null; |
| 27 | - const [scheme, value] = h.split(" ", 2); | |
| 29 | + const [scheme, value] = h.split(/\s+/, 2); | |
| 28 | 30 | if (!value) return null; |
| 29 | 31 | if (/^bearer$/i.test(scheme!) || /^token$/i.test(scheme!)) return value.trim(); |
| 30 | 32 | if (/^basic$/i.test(scheme!)) { |
| 31 | - const decoded = new TextDecoder().decode(fromB64(value.trim())); | |
| 33 | + let bytes: Uint8Array; | |
| 34 | + try { | |
| 35 | + bytes = fromB64(value.trim()); | |
| 36 | + } catch { | |
| 37 | + throw new HttpError(401, "malformed basic credentials"); | |
| 38 | + } | |
| 39 | + const decoded = new TextDecoder().decode(bytes); | |
| 32 | 40 | const idx = decoded.indexOf(":"); |
| 33 | 41 | const pass = idx >= 0 ? decoded.slice(idx + 1) : decoded; |
| 34 | 42 | const user = idx >= 0 ? decoded.slice(0, idx) : ""; |
| @@ -119,11 +127,16 @@ | ||
| 119 | 127 | const p = c.get("principal"); |
| 120 | 128 | const dir = c.get("dir"); |
| 121 | 129 | if (p.type === "repo-token") { |
| 122 | - if (p.repoId !== repo.id) return null; | |
| 130 | + // Like GITHUB_TOKEN, a job or sandbox token reads other repositories only when they are public. | |
| 131 | + if (p.repoId !== repo.id) return repo.visibility === "public" ? { id: "anonymous", login: "anonymous", kind: "user", role: "read" } : null; | |
| 123 | 132 | const stub = c.env.REPO.get(c.env.REPO.idFromName(repo.id)); |
| 124 | 133 | const auth = await stub.runnerAuth(p.token); |
| 125 | 134 | if (!auth) return null; |
| 126 | - if (auth.kind === "job") return { id: `job:${auth.jobId}`, login: "diffinery-actions[bot]", kind: "system", role: "read" }; | |
| 135 | + // A job token can do what the workflow's `permissions:` grant, checked per request in | |
| 136 | + // loadRepo. Runs an agent started act as an agent, so its guardrails still apply. | |
| 137 | + if (auth.kind === "job") { | |
| 138 | + return { id: `job:${auth.jobId}`, login: "diffinery-actions[bot]", kind: auth.actorKind === "agent" ? "agent" : "system", role: auth.role, permissions: auth.permissions }; | |
| 139 | + } | |
| 127 | 140 | const cap: Role = ROLE_LEVEL[auth.caller.role] > ROLE_LEVEL.write ? "write" : auth.caller.role; |
| 128 | 141 | return { ...auth.caller, role: cap }; |
| 129 | 142 | } |
modifiedapps/diffinery/src/docs/api-meta.ts+6−1View file
| @@ -974,7 +974,12 @@ | ||
| 974 | 974 | access: "read", |
| 975 | 975 | query: { format: ["string", "`json` for an array of `{ t, text }` lines."], after: ["integer", "Only lines after this sequence number (for following a live log)."] }, |
| 976 | 976 | }, |
| 977 | - "GET /repos/{owner}/{repo}/actions/runs/{id}/artifacts": { summary: "List a run's artifacts", description: "Each with an `archive_download_url` (a `.tar.gz`).", access: "read" }, | |
| 977 | + "GET /repos/{owner}/{repo}/actions/runs/{runId}/artifacts": { | |
| 978 | + summary: "List a run's artifacts", | |
| 979 | + description: "Each with an `archive_download_url` (a zip). Expired artifacts are listed with `expired: true`; their content is gone.", | |
| 980 | + access: "read", | |
| 981 | + query: { name: ["string", "Only artifacts with this name."] }, | |
| 982 | + }, | |
| 978 | 983 | "GET /repos/{owner}/{repo}/actions/secrets": { summary: "List repository secrets", description: "Names and update times only; values are never returned.", access: "admin" }, |
| 979 | 984 | "PUT /repos/{owner}/{repo}/actions/secrets/{name}": { |
| 980 | 985 | summary: "Set a repository secret", |
modifiedapps/diffinery/src/env.ts+2−0View file
| @@ -26,6 +26,8 @@ | ||
| 26 | 26 | ACCESS_TEAM_DOMAIN: string; |
| 27 | 27 | /** Access application audience tag. */ |
| 28 | 28 | ACCESS_AUD: string; |
| 29 | + /** Artifact storage per repository in bytes (live artifacts plus uploads in progress); default 10 GiB. */ | |
| 30 | + ARTIFACT_QUOTA_BYTES?: string; | |
| 29 | 31 | /** Local development only: act as this email when Access is not configured. */ |
| 30 | 32 | DEV_AUTH?: string; |
| 31 | 33 |
addedapps/diffinery/src/http/actions-runtime.ts+638−0View file
| @@ -1,0 +1,638 @@ | ||
| 1 | +import { Hono } from "hono"; | |
| 2 | +import type { Env } from "../env.js"; | |
| 3 | +import type { AppContext, AppVars } from "../auth/principal.js"; | |
| 4 | +import { blockKey, chunkKey, compose, deleteKeys, finalKey, listStaged, parseBlockList, parseRange, sized, stagingPrefix, type BlobBucket, type ComposeStep } from "../lib/blob-store.js"; | |
| 5 | +import { signBlobUrl, verifyBlobUrl, verifyRuntimeToken, type BlobRef, type RuntimeClaims } from "../lib/runtime-auth.js"; | |
| 6 | +import { HttpError, newId } from "../lib/util.js"; | |
| 7 | +import { MAX_UPLOAD_PART, type ArtifactEntry, type CacheEntry } from "../repo/runtime.js"; | |
| 8 | +import { loadRepo, repoStub } from "./common.js"; | |
| 9 | + | |
| 10 | +/** | |
| 11 | + * GitHub's Actions runtime services, so the official actions and the toolkit | |
| 12 | + * (`@actions/cache`, `@actions/artifact`, buildkit's `type=gha` cache) work | |
| 13 | + * against Diffinery unchanged: | |
| 14 | + * | |
| 15 | + * - Twirp `CacheService` (cache v2) and `ArtifactService` (artifacts v4) at | |
| 16 | + * `ACTIONS_RESULTS_URL`, and the legacy cache REST API at `ACTIONS_CACHE_URL`. | |
| 17 | + * All authenticate with the job's ACTIONS_RUNTIME_TOKEN and work only while | |
| 18 | + * that job runs (repo/runtime.ts decides what may be read or written). | |
| 19 | + * - Signed blob URLs that speak the parts of Azure Blob Storage the clients use | |
| 20 | + * (Put Block, Put Block List, Put Blob, ranged GET), backed by R2. | |
| 21 | + * - The REST API's artifacts and caches endpoints (`actionsRest`, under /api/v3). | |
| 22 | + */ | |
| 23 | +export const actionsRuntime = new Hono<{ Bindings: Env; Variables: AppVars }>(); | |
| 24 | +export const actionsRest = new Hono<{ Bindings: Env; Variables: AppVars }>(); | |
| 25 | + | |
| 26 | +const READ_URL_TTL_S = 3600; | |
| 27 | +const ZIP = "application/zip"; | |
| 28 | + | |
| 29 | +const bucket = (env: Env) => env.BUCKET as unknown as BlobBucket; | |
| 30 | +const origin = (env: Env) => new URL(env.PUBLIC_URL).origin; | |
| 31 | +const nowS = () => Math.floor(Date.now() / 1000); | |
| 32 | + | |
| 33 | +async function runtimeClaims(c: AppContext): Promise<RuntimeClaims> { | |
| 34 | + const m = /^Bearer\s+(\S+)$/i.exec(c.req.header("authorization") ?? ""); | |
| 35 | + const claims = m ? await verifyRuntimeToken(c.env.MASTER_KEY, m[1]!) : null; | |
| 36 | + if (!claims) throw new HttpError(401, "invalid or expired runtime token"); | |
| 37 | + return claims; | |
| 38 | +} | |
| 39 | + | |
| 40 | +/** Status and message of an error thrown here or inside the Durable Object. */ | |
| 41 | +function failure(e: unknown): { status: number; message: string } { | |
| 42 | + if (e instanceof HttpError) return { status: e.status, message: e.detail }; | |
| 43 | + const msg = (e as Error)?.message ?? String(e); | |
| 44 | + const m = /^(\d{3}): ([\s\S]*)$/.exec(msg); | |
| 45 | + if (m) return { status: Number(m[1]), message: m[2]! }; | |
| 46 | + console.error(e); | |
| 47 | + return { status: 500, message: "internal error" }; | |
| 48 | +} | |
| 49 | + | |
| 50 | +const blobName = (a: ArtifactEntry) => (a.mimeType === ZIP ? `${a.name}.zip` : a.name); | |
| 51 | +const readUrl = (env: Env, b: BlobRef, fileName?: string) => signBlobUrl(env.MASTER_KEY, origin(env), b, "r", nowS() + READ_URL_TTL_S, fileName); | |
| 52 | + | |
| 53 | +/** | |
| 54 | + * A cache lookup whose object is really there. An entry whose object went missing | |
| 55 | + * (or was replaced after it was committed) is dropped, and the lookup tries again. | |
| 56 | + */ | |
| 57 | +async function liveCacheHit(env: Env, claims: RuntimeClaims, keys: string[], version: string): Promise<CacheEntry | null> { | |
| 58 | + const stub = repoStub(env, claims.repo); | |
| 59 | + for (let i = 0; i < 5; i++) { | |
| 60 | + const hit = await stub.rtCacheLookup(claims.job, keys, version); | |
| 61 | + if (!hit) return null; | |
| 62 | + const head = await env.BUCKET.head(finalKey({ repo: claims.repo, kind: "cache", id: hit.id })); | |
| 63 | + if (head && (!hit.etag || head.etag === hit.etag)) return hit; | |
| 64 | + await stub.rtCacheMissing(hit.id); | |
| 65 | + } | |
| 66 | + return null; | |
| 67 | +} | |
| 68 | + | |
| 69 | +/** Composes an upload's staged pieces, each step in a Durable Object call of its own (lib/blob-store.ts). */ | |
| 70 | +function composeUpload(env: Env, b: BlobRef, segments: Array<{ key: string; size: number }>, contentType: string): Promise<number> { | |
| 71 | + const stub = repoStub(env, b.repo); | |
| 72 | + return compose(bucket(env), finalKey(b), segments, contentType, { | |
| 73 | + scratch: stagingPrefix(b), | |
| 74 | + run: (step: ComposeStep) => stub.rtComposeStep({ kind: b.kind, id: b.id }, step, contentType), | |
| 75 | + }); | |
| 76 | +} | |
| 77 | + | |
| 78 | +/** A request's declared length; null without Content-Length. */ | |
| 79 | +function declaredLength(c: AppContext): number | null { | |
| 80 | + const v = c.req.header("content-length"); | |
| 81 | + if (v === undefined) return null; | |
| 82 | + if (!/^\d{1,15}$/.test(v)) throw new HttpError(400, "invalid Content-Length"); | |
| 83 | + return Number(v); | |
| 84 | +} | |
| 85 | + | |
| 86 | +/** A small text body, refused past `max` bytes (with or without Content-Length). */ | |
| 87 | +async function smallText(c: AppContext, max: number): Promise<string> { | |
| 88 | + const len = declaredLength(c); | |
| 89 | + if (len !== null && len > max) throw new HttpError(413, "the request body is too large"); | |
| 90 | + const body = c.req.raw.body; | |
| 91 | + if (!body) return ""; | |
| 92 | + const reader = body.getReader(); | |
| 93 | + const chunks: Uint8Array[] = []; | |
| 94 | + let n = 0; | |
| 95 | + for (;;) { | |
| 96 | + const { done, value } = await reader.read(); | |
| 97 | + if (done) break; | |
| 98 | + n += value.length; | |
| 99 | + if (n > max) { | |
| 100 | + await reader.cancel().catch(() => {}); | |
| 101 | + throw new HttpError(413, "the request body is too large"); | |
| 102 | + } | |
| 103 | + chunks.push(value); | |
| 104 | + } | |
| 105 | + const out = new Uint8Array(n); | |
| 106 | + let at = 0; | |
| 107 | + for (const ch of chunks) { | |
| 108 | + out.set(ch, at); | |
| 109 | + at += ch.length; | |
| 110 | + } | |
| 111 | + return new TextDecoder().decode(out); | |
| 112 | +} | |
| 113 | + | |
| 114 | +// ---------------------------------------------------------------- Twirp services | |
| 115 | + | |
| 116 | +const TWIRP_CODES: Record<number, string> = { 400: "invalid_argument", 401: "unauthenticated", 403: "permission_denied", 404: "not_found", 409: "already_exists", 412: "failed_precondition", 429: "resource_exhausted" }; | |
| 117 | + | |
| 118 | +type TwirpRequest = Record<string, unknown>; | |
| 119 | + | |
| 120 | +/** A request field by its proto name or its JSON (camelCase) name. */ | |
| 121 | +function field(req: TwirpRequest, name: string): unknown { | |
| 122 | + return req[name] ?? req[name.replace(/_([a-z])/g, (_, ch: string) => ch.toUpperCase())]; | |
| 123 | +} | |
| 124 | + | |
| 125 | +function str(req: TwirpRequest, name: string, max = 1024): string { | |
| 126 | + const v = field(req, name) ?? ""; | |
| 127 | + if (typeof v !== "string" || v.length > max) throw new HttpError(400, `${name} must be a string of at most ${max} characters`); | |
| 128 | + return v; | |
| 129 | +} | |
| 130 | + | |
| 131 | +/** int64 values arrive as strings (protobuf JSON) or numbers (Go clients). */ | |
| 132 | +function int64(req: TwirpRequest, name: string): number | undefined { | |
| 133 | + const v = field(req, name); | |
| 134 | + if (v === undefined || v === null || v === "") return undefined; | |
| 135 | + const n = typeof v === "number" ? v : typeof v === "string" && /^\d{1,16}$/.test(v) ? Number(v) : NaN; | |
| 136 | + if (!Number.isSafeInteger(n) || n < 0) throw new HttpError(400, `${name} must be a non-negative integer`); | |
| 137 | + return n; | |
| 138 | +} | |
| 139 | + | |
| 140 | +function twirp(service: "CacheService" | "ArtifactService", method: string, fn: (c: AppContext, claims: RuntimeClaims, req: TwirpRequest) => Promise<Record<string, unknown>>) { | |
| 141 | + actionsRuntime.post(`/twirp/github.actions.results.api.v1.${service}/${method}`, async (c) => { | |
| 142 | + try { | |
| 143 | + const claims = await runtimeClaims(c); | |
| 144 | + let req: unknown; | |
| 145 | + try { | |
| 146 | + req = await c.req.json(); | |
| 147 | + } catch { | |
| 148 | + throw new HttpError(400, "the request body must be JSON"); | |
| 149 | + } | |
| 150 | + if (!req || typeof req !== "object" || Array.isArray(req)) throw new HttpError(400, "the request body must be a JSON object"); | |
| 151 | + return c.json(await fn(c, claims, req as TwirpRequest)); | |
| 152 | + } catch (e) { | |
| 153 | + const { status, message } = failure(e); | |
| 154 | + return c.json({ code: TWIRP_CODES[status] ?? "internal", msg: message }, status as 400); | |
| 155 | + } | |
| 156 | + }); | |
| 157 | +} | |
| 158 | + | |
| 159 | +/** Reads name the run; the job may be the one that uploaded (download-artifact sends the artifact's ids). */ | |
| 160 | +function ownRun(claims: RuntimeClaims, req: TwirpRequest): void { | |
| 161 | + const run = field(req, "workflow_run_backend_id"); | |
| 162 | + if (run !== undefined && run !== claims.run) throw new HttpError(403, "the request names another workflow run"); | |
| 163 | +} | |
| 164 | + | |
| 165 | +/** Writes name the run and job; they must be the token's own. */ | |
| 166 | +function ownJob(claims: RuntimeClaims, req: TwirpRequest): void { | |
| 167 | + const run = field(req, "workflow_run_backend_id"); | |
| 168 | + const job = field(req, "workflow_job_run_backend_id"); | |
| 169 | + if ((run !== undefined && run !== claims.run) || (job !== undefined && job !== claims.job)) throw new HttpError(403, "the request names another workflow run or job"); | |
| 170 | +} | |
| 171 | + | |
| 172 | +twirp("CacheService", "CreateCacheEntry", async (c, claims, req) => { | |
| 173 | + const reserved = await repoStub(c.env, claims.repo).rtCacheReserve(claims.job, str(req, "key", 512), str(req, "version", 128)); | |
| 174 | + if (!reserved) throw new HttpError(409, "a cache entry with this key and version already exists"); | |
| 175 | + const url = await signBlobUrl(c.env.MASTER_KEY, origin(c.env), { repo: claims.repo, kind: "cache", id: reserved.id }, "w", claims.exp); | |
| 176 | + return { ok: true, signed_upload_url: url }; | |
| 177 | +}); | |
| 178 | + | |
| 179 | +twirp("CacheService", "FinalizeCacheEntryUpload", async (c, claims, req) => { | |
| 180 | + const stub = repoStub(c.env, claims.repo); | |
| 181 | + const entry = await stub.rtCacheReservation(claims.job, { key: str(req, "key", 512), version: str(req, "version", 128) }); | |
| 182 | + if (!entry) throw new HttpError(404, "no upload of this cache entry is in progress"); | |
| 183 | + const obj = await c.env.BUCKET.head(finalKey({ repo: claims.repo, kind: "cache", id: entry.id })); | |
| 184 | + if (!obj) return { ok: false, entry_id: "0", message: "the cache archive was not uploaded" }; | |
| 185 | + const size = int64(req, "size_bytes"); | |
| 186 | + if (size !== undefined && size !== obj.size) return { ok: false, entry_id: "0", message: `the uploaded archive has ${obj.size} bytes, not ${size}` }; | |
| 187 | + const done = await stub.rtCacheCommit(claims.job, entry.id, obj.size, obj.etag); | |
| 188 | + return { ok: true, entry_id: String(done.id) }; | |
| 189 | +}); | |
| 190 | + | |
| 191 | +twirp("CacheService", "GetCacheEntryDownloadURL", async (c, claims, req) => { | |
| 192 | + const restore = field(req, "restore_keys"); | |
| 193 | + if (restore !== undefined && (!Array.isArray(restore) || restore.length > 20 || restore.some((k) => typeof k !== "string" || k.length > 512))) throw new HttpError(400, "restore_keys must be a list of at most 20 keys"); | |
| 194 | + const keys = [...new Set([str(req, "key", 512), ...((restore as string[] | undefined) ?? [])].filter(Boolean))]; | |
| 195 | + if (!keys.length) throw new HttpError(400, "key is required"); | |
| 196 | + const hit = await liveCacheHit(c.env, claims, keys, str(req, "version", 128)); | |
| 197 | + if (!hit) return { ok: false, signed_download_url: "", matched_key: "" }; | |
| 198 | + return { ok: true, signed_download_url: await readUrl(c.env, { repo: claims.repo, kind: "cache", id: hit.id }), matched_key: hit.key }; | |
| 199 | +}); | |
| 200 | + | |
| 201 | +const artifactName = (req: TwirpRequest) => str(req, "name", 256); | |
| 202 | + | |
| 203 | +twirp("ArtifactService", "CreateArtifact", async (c, claims, req) => { | |
| 204 | + ownJob(claims, req); | |
| 205 | + const mime = field(req, "mime_type"); | |
| 206 | + const mimeType = typeof mime === "string" && /^[\w.+-]{1,64}\/[\w.+-]{1,64}$/.test(mime) ? mime : ZIP; | |
| 207 | + const expires = field(req, "expires_at"); | |
| 208 | + const { id } = await repoStub(c.env, claims.repo).rtArtifactCreate(claims.job, artifactName(req), mimeType, typeof expires === "string" ? expires : undefined); | |
| 209 | + const url = await signBlobUrl(c.env.MASTER_KEY, origin(c.env), { repo: claims.repo, kind: "artifact", id }, "w", claims.exp); | |
| 210 | + return { ok: true, signed_upload_url: url }; | |
| 211 | +}); | |
| 212 | + | |
| 213 | +twirp("ArtifactService", "FinalizeArtifact", async (c, claims, req) => { | |
| 214 | + ownJob(claims, req); | |
| 215 | + const stub = repoStub(c.env, claims.repo); | |
| 216 | + const name = artifactName(req); | |
| 217 | + const upload = await stub.rtArtifactUpload(claims.job, name); | |
| 218 | + if (!upload) throw new HttpError(404, `no upload of artifact '${name}' is in progress`); | |
| 219 | + const obj = await c.env.BUCKET.head(finalKey({ repo: claims.repo, kind: "artifact", id: upload.id })); | |
| 220 | + if (!obj) throw new HttpError(412, "the artifact's content was not uploaded"); | |
| 221 | + const size = int64(req, "size"); | |
| 222 | + if (size !== undefined && size !== obj.size) throw new HttpError(412, `the uploaded artifact has ${obj.size} bytes, not ${size}`); | |
| 223 | + const hash = field(req, "hash"); | |
| 224 | + const digest = typeof hash === "string" && /^sha256:[0-9a-f]{64}$/.test(hash) ? hash : undefined; | |
| 225 | + const done = await stub.rtArtifactFinalize(claims.job, name, obj.size, digest, obj.etag); | |
| 226 | + return { ok: true, artifact_id: String(done.id) }; | |
| 227 | +}); | |
| 228 | + | |
| 229 | +twirp("ArtifactService", "ListArtifacts", async (c, claims, req) => { | |
| 230 | + ownRun(claims, req); | |
| 231 | + const nameFilter = field(req, "name_filter"); | |
| 232 | + const list = await repoStub(c.env, claims.repo).rtJobArtifacts(claims.job, { name: typeof nameFilter === "string" ? nameFilter : undefined, id: int64(req, "id_filter") }); | |
| 233 | + return { | |
| 234 | + artifacts: list.map((a) => ({ | |
| 235 | + workflow_run_backend_id: a.runId, | |
| 236 | + workflow_job_run_backend_id: a.jobId, | |
| 237 | + database_id: String(a.id), | |
| 238 | + name: a.name, | |
| 239 | + size: String(a.size), | |
| 240 | + created_at: a.createdAt, | |
| 241 | + ...(a.digest ? { digest: a.digest } : {}), | |
| 242 | + })), | |
| 243 | + }; | |
| 244 | +}); | |
| 245 | + | |
| 246 | +twirp("ArtifactService", "GetSignedArtifactURL", async (c, claims, req) => { | |
| 247 | + ownRun(claims, req); | |
| 248 | + const name = artifactName(req); | |
| 249 | + const a = (await repoStub(c.env, claims.repo).rtJobArtifacts(claims.job, { name }))[0]; | |
| 250 | + if (!a) throw new HttpError(404, `artifact '${name}' not found`); | |
| 251 | + return { signed_url: await readUrl(c.env, { repo: claims.repo, kind: "artifact", id: a.id }, blobName(a)) }; | |
| 252 | +}); | |
| 253 | + | |
| 254 | +twirp("ArtifactService", "DeleteArtifact", async (c, claims, req) => { | |
| 255 | + ownJob(claims, req); | |
| 256 | + const name = artifactName(req); | |
| 257 | + const deleted = await repoStub(c.env, claims.repo).rtArtifactDeleteByName(claims.job, name); | |
| 258 | + if (!deleted) throw new HttpError(404, `artifact '${name}' not found`); | |
| 259 | + return { ok: true, artifact_id: String(deleted.id) }; | |
| 260 | +}); | |
| 261 | + | |
| 262 | +// ---------------------------------------------------------------- legacy cache API (ACTIONS_CACHE_URL) | |
| 263 | + | |
| 264 | +const V1 = "/_runtime/_apis/artifactcache"; | |
| 265 | + | |
| 266 | +/** Errors in the shape the legacy API used (clients look at `typeKey` and `message`). */ | |
| 267 | +function v1(fn: (c: AppContext, claims: RuntimeClaims) => Promise<Response>) { | |
| 268 | + return async (c: AppContext) => { | |
| 269 | + try { | |
| 270 | + return await fn(c, await runtimeClaims(c)); | |
| 271 | + } catch (e) { | |
| 272 | + const { status, message } = failure(e); | |
| 273 | + const typeKey = status === 409 ? "ArtifactCacheItemAlreadyExistsException" : status === 404 ? "ArtifactCacheNotFoundException" : "ArtifactCacheException"; | |
| 274 | + return c.json({ $id: "1", innerException: null, message, typeName: `Microsoft.Azure.DevOps.ArtifactCache.WebApi.${typeKey}, Microsoft.Azure.DevOps.ArtifactCache.WebApi`, typeKey, errorCode: 0, eventId: 3000 }, status as 400); | |
| 275 | + } | |
| 276 | + }; | |
| 277 | +} | |
| 278 | + | |
| 279 | +const v1Cache = (e: CacheEntry) => ({ cacheKey: e.key, scope: e.scope, cacheVersion: e.version, creationTime: e.createdAt }); | |
| 280 | + | |
| 281 | +actionsRuntime.get( | |
| 282 | + `${V1}/cache`, | |
| 283 | + v1(async (c, claims) => { | |
| 284 | + const keys = (c.req.query("keys") ?? "").split(",").map((k) => k.trim()).filter(Boolean); | |
| 285 | + if (!keys.length || keys.length > 20 || keys.some((k) => k.length > 512)) throw new HttpError(400, "keys must list 1 to 20 cache keys"); | |
| 286 | + const hit = await liveCacheHit(c.env, claims, keys, (c.req.query("version") ?? "").slice(0, 128)); | |
| 287 | + if (!hit) return c.body(null, 204); | |
| 288 | + return c.json({ ...v1Cache(hit), archiveLocation: await readUrl(c.env, { repo: claims.repo, kind: "cache", id: hit.id }) }); | |
| 289 | + }), | |
| 290 | +); | |
| 291 | + | |
| 292 | +actionsRuntime.get( | |
| 293 | + `${V1}/caches`, | |
| 294 | + v1(async (c, claims) => { | |
| 295 | + const stub = repoStub(c.env, claims.repo); | |
| 296 | + const { scopes } = await stub.rtJob(claims.job); | |
| 297 | + const key = (c.req.query("key") ?? "").slice(0, 512); | |
| 298 | + const { caches } = await stub.rtListCaches({ key: key || undefined, perPage: 100 }); | |
| 299 | + const visible = caches.filter((e) => scopes.includes(e.scope)); | |
| 300 | + return c.json({ totalCount: visible.length, artifactCaches: visible.map(v1Cache) }); | |
| 301 | + }), | |
| 302 | +); | |
| 303 | + | |
| 304 | +actionsRuntime.post( | |
| 305 | + `${V1}/caches`, | |
| 306 | + v1(async (c, claims) => { | |
| 307 | + const b = (await c.req.json().catch(() => null)) as { key?: unknown; version?: unknown; cacheSize?: unknown } | null; | |
| 308 | + if (!b || typeof b.key !== "string" || typeof b.version !== "string") throw new HttpError(400, "key and version are required"); | |
| 309 | + if (typeof b.cacheSize === "number" && b.cacheSize > 10 * 1024 ** 3) throw new HttpError(400, "the cache is larger than the 10 GB limit"); | |
| 310 | + const reserved = await repoStub(c.env, claims.repo).rtCacheReserve(claims.job, b.key, b.version); | |
| 311 | + if (!reserved) throw new HttpError(409, `Cache already exists. Key: ${b.key}, Version: ${b.version}`); | |
| 312 | + return c.json({ cacheId: reserved.id }, 201); | |
| 313 | + }), | |
| 314 | +); | |
| 315 | + | |
| 316 | +const cacheIdParam = (c: AppContext) => { | |
| 317 | + const id = Number(c.req.param("id")); | |
| 318 | + if (!Number.isSafeInteger(id) || id < 1) throw new HttpError(404, "cache not found"); | |
| 319 | + return id; | |
| 320 | +}; | |
| 321 | + | |
| 322 | +actionsRuntime.patch( | |
| 323 | + `${V1}/caches/:id`, | |
| 324 | + v1(async (c, claims) => { | |
| 325 | + const id = cacheIdParam(c); | |
| 326 | + const m = /^bytes (\d{1,15})-(\d{1,15})\/(\*|\d+)$/.exec(c.req.header("content-range") ?? ""); | |
| 327 | + if (!m) throw new HttpError(400, "Content-Range must be bytes <start>-<end>/*"); | |
| 328 | + const start = Number(m[1]); | |
| 329 | + const length = Number(m[2]) - start + 1; | |
| 330 | + if (length < 1) throw new HttpError(400, "invalid Content-Range"); | |
| 331 | + const declared = declaredLength(c); | |
| 332 | + if (declared !== null && declared !== length) throw new HttpError(400, `the chunk has ${declared} bytes but Content-Range says ${length}`); | |
| 333 | + const stub = repoStub(c.env, claims.repo); | |
| 334 | + if (!(await stub.rtCacheReservation(claims.job, { id }))) throw new HttpError(404, "no upload of this cache entry is in progress"); | |
| 335 | + // Counted against the budget before anything is stored; the body streams to R2 | |
| 336 | + // (@actions/cache sends 32 MiB chunks, four at a time, without Content-Length). | |
| 337 | + if (!c.req.raw.body) throw new HttpError(400, "the chunk is empty"); | |
| 338 | + await stub.rtStage("cache", id, length); | |
| 339 | + try { | |
| 340 | + await c.env.BUCKET.put(chunkKey({ repo: claims.repo, kind: "cache", id }, start), sized(c.req.raw.body, length)); | |
| 341 | + } catch { | |
| 342 | + await stub.rtUnstage("cache", id, length); | |
| 343 | + throw new HttpError(400, `the chunk does not have the ${length} bytes Content-Range says`); | |
| 344 | + } | |
| 345 | + return c.body(null, 204); | |
| 346 | + }), | |
| 347 | +); | |
| 348 | + | |
| 349 | +actionsRuntime.post( | |
| 350 | + `${V1}/caches/:id`, | |
| 351 | + v1(async (c, claims) => { | |
| 352 | + const id = cacheIdParam(c); | |
| 353 | + const b = (await c.req.json().catch(() => null)) as { size?: unknown } | null; | |
| 354 | + const size = b?.size; | |
| 355 | + if (typeof size !== "number" || !Number.isSafeInteger(size) || size < 0) throw new HttpError(400, "size is required"); | |
| 356 | + const stub = repoStub(c.env, claims.repo); | |
| 357 | + if (!(await stub.rtCacheReservation(claims.job, { id }))) throw new HttpError(404, "no upload of this cache entry is in progress"); | |
| 358 | + const ref: BlobRef = { repo: claims.repo, kind: "cache", id }; | |
| 359 | + const staged = await listStaged(bucket(c.env), ref); | |
| 360 | + // Chunks are keyed by their offset; together they must cover [0, size) exactly. | |
| 361 | + const chunks = [...staged].flatMap(([key, n]) => { | |
| 362 | + const off = /\/o-(\d{16})$/.exec(key); | |
| 363 | + return off ? [{ key, size: n, offset: Number(off[1]) }] : []; | |
| 364 | + }); | |
| 365 | + chunks.sort((x, y) => x.offset - y.offset); | |
| 366 | + let at = 0; | |
| 367 | + for (const ch of chunks) { | |
| 368 | + if (ch.offset !== at) throw new HttpError(400, `the uploaded chunks do not cover byte ${at} exactly`); | |
| 369 | + at += ch.size; | |
| 370 | + } | |
| 371 | + if (at !== size) throw new HttpError(400, `${at} bytes were uploaded, not ${size}`); | |
| 372 | + await stub.rtCheckCommitSize("cache", id, size); | |
| 373 | + await composeUpload(c.env, ref, chunks, "application/octet-stream"); | |
| 374 | + await deleteKeys(bucket(c.env), [...staged.keys()]); | |
| 375 | + const obj = await c.env.BUCKET.head(finalKey(ref)); | |
| 376 | + if (!obj) throw new HttpError(500, "the composed cache archive is missing"); | |
| 377 | + await stub.rtCacheCommit(claims.job, id, obj.size, obj.etag); | |
| 378 | + return c.body(null, 204); | |
| 379 | + }), | |
| 380 | +); | |
| 381 | + | |
| 382 | +// ---------------------------------------------------------------- signed blob URLs (Azure Blob subset on R2) | |
| 383 | + | |
| 384 | +const BLOB_PATH = "/_runtime/blob/:repo/:kind/:id/:file?"; | |
| 385 | + | |
| 386 | +function blobRef(c: AppContext): BlobRef | null { | |
| 387 | + const { repo, kind, id } = c.req.param() as { repo: string; kind: string; id: string }; | |
| 388 | + if (!/^[A-Za-z0-9_-]{1,64}$/.test(repo) || (kind !== "cache" && kind !== "artifact") || !/^\d{1,15}$/.test(id)) return null; | |
| 389 | + return { repo, kind, id: Number(id) }; | |
| 390 | +} | |
| 391 | + | |
| 392 | +/** Headers every Azure Blob response carries. */ | |
| 393 | +const azureHeaders = (extra: Record<string, string> = {}): Record<string, string> => ({ "x-ms-request-id": newId(), "x-ms-version": "2025-01-05", date: new Date().toUTCString(), ...extra }); | |
| 394 | + | |
| 395 | +function azureError(status: number, code: string, message: string): Response { | |
| 396 | + const xml = `<?xml version="1.0" encoding="utf-8"?><Error><Code>${code}</Code><Message>${message.replace(/[<>&]/g, "")}</Message></Error>`; | |
| 397 | + return new Response(xml, { status, headers: azureHeaders({ "content-type": "application/xml", "x-ms-error-code": code }) }); | |
| 398 | +} | |
| 399 | + | |
| 400 | +const created = (etag?: string) => new Response(null, { status: 201, headers: azureHeaders({ "x-ms-request-server-encrypted": "true", ...(etag ? { etag, "last-modified": new Date().toUTCString() } : {}) }) }); | |
| 401 | + | |
| 402 | +/** | |
| 403 | + * Stores a block or blob body as it streams in. Like Azure, a length is required | |
| 404 | + * (411 without one); the bytes are counted against the repository's budget first. | |
| 405 | + */ | |
| 406 | +async function putUpload(c: AppContext, b: BlobRef, key: string, contentType?: string): Promise<{ etag: string; httpEtag: string } | Response> { | |
| 407 | + const len = declaredLength(c); | |
| 408 | + if (len === null) return azureError(411, "MissingContentLengthHeader", "Content-Length is required."); | |
| 409 | + if (len > MAX_UPLOAD_PART) return azureError(413, "RequestBodyTooLarge", `A single request may carry at most ${MAX_UPLOAD_PART} bytes.`); | |
| 410 | + const stub = repoStub(c.env, b.repo); | |
| 411 | + await stub.rtStage(b.kind, b.id, len); | |
| 412 | + const body = len > 0 && c.req.raw.body ? sized(c.req.raw.body, len) : new Uint8Array(0); | |
| 413 | + try { | |
| 414 | + const put = await c.env.BUCKET.put(key, body, contentType ? { httpMetadata: { contentType } } : undefined); | |
| 415 | + return { etag: put.etag, httpEtag: put.httpEtag }; | |
| 416 | + } catch (e) { | |
| 417 | + await stub.rtUnstage(b.kind, b.id, len); | |
| 418 | + throw e; | |
| 419 | + } | |
| 420 | +} | |
| 421 | + | |
| 422 | +const BLOCK_LIST_MAX = 8 * 1024 * 1024; | |
| 423 | + | |
| 424 | +actionsRuntime.put(BLOB_PATH, async (c) => { | |
| 425 | + const b = blobRef(c); | |
| 426 | + if (!b) return azureError(404, "ResourceNotFound", "The specified resource does not exist."); | |
| 427 | + const q = new URL(c.req.url).searchParams; | |
| 428 | + if (!(await verifyBlobUrl(c.env.MASTER_KEY, b, "w", q))) return azureError(403, "AuthenticationFailed", "The upload URL is invalid or has expired."); | |
| 429 | + try { | |
| 430 | + const comp = q.get("comp"); | |
| 431 | + if (comp === "block") { | |
| 432 | + const blockId = q.get("blockid") ?? ""; | |
| 433 | + if (!blockId || blockId.length > 1024) return azureError(400, "InvalidQueryParameterValue", "blockid is required."); | |
| 434 | + const put = await putUpload(c, b, blockKey(b, blockId)); | |
| 435 | + return put instanceof Response ? put : created(); | |
| 436 | + } | |
| 437 | + if (comp !== "blocklist" && comp !== null) return azureError(400, "InvalidQueryParameterValue", `comp=${comp} is not supported.`); | |
| 438 | + const stub = repoStub(c.env, b.repo); | |
| 439 | + const writable = b.kind === "cache" ? await stub.rtCacheWritable(b.id) : await stub.rtArtifactWritable(b.id); | |
| 440 | + if (!writable) return azureError(409, "BlobImmutable", "This upload is finished or its job is no longer running."); | |
| 441 | + const contentType = c.req.header("x-ms-blob-content-type") ?? "application/octet-stream"; | |
| 442 | + if (comp === null) { | |
| 443 | + // Put Blob: the whole content in one request. | |
| 444 | + if ((c.req.header("x-ms-blob-type") ?? "BlockBlob") !== "BlockBlob") return azureError(400, "InvalidHeaderValue", "Only block blobs are supported."); | |
| 445 | + const put = await putUpload(c, b, finalKey(b), contentType); | |
| 446 | + return put instanceof Response ? put : created(put.httpEtag); | |
| 447 | + } | |
| 448 | + // Put Block List: concatenate the staged blocks in the order given. | |
| 449 | + const ids = parseBlockList(await smallText(c, BLOCK_LIST_MAX)); | |
| 450 | + if (ids.length > 50_000) return azureError(400, "InvalidBlockList", "Too many blocks."); | |
| 451 | + const staged = await listStaged(bucket(c.env), b); | |
| 452 | + const segments: Array<{ key: string; size: number }> = []; | |
| 453 | + for (const id of ids) { | |
| 454 | + const key = blockKey(b, id); | |
| 455 | + const size = staged.get(key); | |
| 456 | + if (size === undefined) return azureError(400, "InvalidBlockList", "The specified block list is invalid."); | |
| 457 | + segments.push({ key, size }); | |
| 458 | + } | |
| 459 | + await stub.rtCheckCommitSize(b.kind, b.id, segments.reduce((n, s) => n + s.size, 0)); | |
| 460 | + await composeUpload(c.env, b, segments, contentType); | |
| 461 | + await deleteKeys(bucket(c.env), [...staged.keys()]); | |
| 462 | + const head = await c.env.BUCKET.head(finalKey(b)); | |
| 463 | + return created(head?.httpEtag ?? `"${newId()}"`); | |
| 464 | + } catch (e) { | |
| 465 | + const { status, message } = failure(e); | |
| 466 | + return azureError(status, status === 404 ? "ResourceNotFound" : status === 413 ? "RequestBodyTooLarge" : status === 409 ? "BlobImmutable" : status < 500 ? "InvalidInput" : "InternalError", message); | |
| 467 | + } | |
| 468 | +}); | |
| 469 | + | |
| 470 | +const serveBlob = async (c: AppContext) => { | |
| 471 | + const b = blobRef(c); | |
| 472 | + if (!b) return azureError(404, "ResourceNotFound", "The specified resource does not exist."); | |
| 473 | + if (!(await verifyBlobUrl(c.env.MASTER_KEY, b, "r", new URL(c.req.url).searchParams))) return azureError(403, "AuthenticationFailed", "The download URL is invalid or has expired."); | |
| 474 | + const stub = repoStub(c.env, b.repo); | |
| 475 | + const headers: Record<string, string> = { "accept-ranges": "bytes", "x-ms-blob-type": "BlockBlob", "content-type": "application/octet-stream" }; | |
| 476 | + let etag: string | null; | |
| 477 | + if (b.kind === "cache") { | |
| 478 | + const e = await stub.rtCacheGet(b.id); | |
| 479 | + if (!e) return azureError(404, "BlobNotFound", "The cache entry does not exist."); | |
| 480 | + etag = e.etag; | |
| 481 | + } else { | |
| 482 | + const a = await stub.rtArtifactGet(b.id); | |
| 483 | + if (!a) return azureError(404, "BlobNotFound", "The artifact does not exist or has expired."); | |
| 484 | + etag = a.etag; | |
| 485 | + const name = blobName(a); | |
| 486 | + headers["content-type"] = a.mimeType; | |
| 487 | + headers["content-disposition"] = `attachment; filename="${name.replace(/[^\w.-]+/g, "_")}"; filename*=UTF-8''${encodeURIComponent(name)}`; | |
| 488 | + if (a.digest) headers["x-ms-meta-digest"] = a.digest; | |
| 489 | + } | |
| 490 | + const key = finalKey(b); | |
| 491 | + const head = await c.env.BUCKET.head(key); | |
| 492 | + if (!head) return azureError(404, "BlobNotFound", "The specified blob does not exist."); | |
| 493 | + // Only the object that was committed: a write that landed after Finalize is not served. | |
| 494 | + if (etag && head.etag !== etag) return azureError(409, "BlobModified", "The blob was modified after it was committed."); | |
| 495 | + Object.assign(headers, { etag: head.httpEtag, "last-modified": head.uploaded.toUTCString() }); | |
| 496 | + const range = parseRange(c.req.header("x-ms-range") ?? c.req.header("range"), head.size); | |
| 497 | + if (range === "invalid") return new Response(null, { status: 416, headers: azureHeaders({ ...headers, "content-range": `bytes */${head.size}` }) }); | |
| 498 | + if (c.req.method === "HEAD") return new Response(null, { status: 200, headers: azureHeaders({ ...headers, "content-length": String(head.size) }) }); | |
| 499 | + if (!range) { | |
| 500 | + const obj = await c.env.BUCKET.get(key, { onlyIf: { etagMatches: head.etag } }); | |
| 501 | + if (!obj || !("body" in obj)) return azureError(404, "BlobNotFound", "The specified blob does not exist."); | |
| 502 | + return new Response(obj.body, { status: 200, headers: azureHeaders({ ...headers, "content-length": String(obj.size) }) }); | |
| 503 | + } | |
| 504 | + const obj = await c.env.BUCKET.get(key, { range: { offset: range.offset, length: range.length }, onlyIf: { etagMatches: head.etag } }); | |
| 505 | + if (!obj || !("body" in obj)) return azureError(404, "BlobNotFound", "The specified blob does not exist."); | |
| 506 | + return new Response(obj.body, { | |
| 507 | + status: 206, | |
| 508 | + headers: azureHeaders({ ...headers, "content-length": String(range.length), "content-range": `bytes ${range.offset}-${range.offset + range.length - 1}/${head.size}` }), | |
| 509 | + }); | |
| 510 | +}; | |
| 511 | +actionsRuntime.get(BLOB_PATH, serveBlob); | |
| 512 | +actionsRuntime.on("HEAD", BLOB_PATH, serveBlob); | |
| 513 | + | |
| 514 | +// ---------------------------------------------------------------- REST: artifacts and caches (/api/v3) | |
| 515 | + | |
| 516 | +const P = (c: AppContext) => ({ owner: c.req.param("owner")!, repo: c.req.param("repo")! }); | |
| 517 | + | |
| 518 | +const paging = (c: AppContext) => { | |
| 519 | + const perPage = Math.min(Math.max(Number(c.req.query("per_page") ?? 30) || 30, 1), 100); | |
| 520 | + const page = Math.max(Number(c.req.query("page") ?? 1) || 1, 1); | |
| 521 | + return { perPage, page }; | |
| 522 | +}; | |
| 523 | + | |
| 524 | +const idParam = (c: AppContext, name: string) => { | |
| 525 | + const n = Number(c.req.param(name)); | |
| 526 | + if (!Number.isSafeInteger(n) || n < 1) throw new HttpError(404, "Not Found"); | |
| 527 | + return n; | |
| 528 | +}; | |
| 529 | + | |
| 530 | +function artifactJson(c: AppContext, repoId: string, a: ArtifactEntry) { | |
| 531 | + const { owner, repo } = P(c); | |
| 532 | + const url = `${c.env.PUBLIC_URL}/api/v3/repos/${owner}/${repo}/actions/artifacts/${a.id}`; | |
| 533 | + return { | |
| 534 | + id: a.id, | |
| 535 | + node_id: `MDg6QXJ0aWZhY3Q${a.id}`, | |
| 536 | + name: a.name, | |
| 537 | + size_in_bytes: a.size, | |
| 538 | + url, | |
| 539 | + archive_download_url: `${url}/zip`, | |
| 540 | + expired: a.expired, | |
| 541 | + digest: a.digest, | |
| 542 | + created_at: a.createdAt, | |
| 543 | + updated_at: a.updatedAt, | |
| 544 | + expires_at: a.expiresAt, | |
| 545 | + workflow_run: { id: a.runNumber, repository_id: repoId, head_repository_id: repoId, head_branch: a.headBranch, head_sha: a.headSha }, | |
| 546 | + }; | |
| 547 | +} | |
| 548 | + | |
| 549 | +const cacheJson = (e: CacheEntry) => ({ id: e.id, ref: e.scope, key: e.key, version: e.version, last_accessed_at: e.lastUsed, created_at: e.createdAt, size_in_bytes: e.size }); | |
| 550 | + | |
| 551 | +actionsRest.get("/repos/:owner/:repo/actions/artifacts", async (c) => { | |
| 552 | + const { repo, stub } = await loadRepo(c, P(c).owner, P(c).repo); | |
| 553 | + const { perPage, page } = paging(c); | |
| 554 | + const res = await stub.rtRepoArtifacts({ name: c.req.query("name") || undefined, page, perPage }); | |
| 555 | + return c.json({ total_count: res.total, artifacts: res.artifacts.map((a) => artifactJson(c, repo.id, a)) }); | |
| 556 | +}); | |
| 557 | + | |
| 558 | +actionsRest.get("/repos/:owner/:repo/actions/runs/:runId/artifacts", async (c) => { | |
| 559 | + const { repo, stub } = await loadRepo(c, P(c).owner, P(c).repo); | |
| 560 | + const runId = c.req.param("runId"); | |
| 561 | + if (!/^[A-Za-z0-9]{1,64}$/.test(runId)) throw new HttpError(404, "Not Found"); | |
| 562 | + const { perPage, page } = paging(c); | |
| 563 | + const all = await stub.rtRunArtifacts(runId, { name: c.req.query("name") || undefined, includeExpired: true }); | |
| 564 | + return c.json({ total_count: all.length, artifacts: all.slice((page - 1) * perPage, page * perPage).map((a) => artifactJson(c, repo.id, a)) }); | |
| 565 | +}); | |
| 566 | + | |
| 567 | +actionsRest.get("/repos/:owner/:repo/actions/artifacts/:id", async (c) => { | |
| 568 | + const { repo, stub } = await loadRepo(c, P(c).owner, P(c).repo); | |
| 569 | + const a = await stub.rtArtifactGet(idParam(c, "id"), { includeExpired: true }); | |
| 570 | + if (!a) throw new HttpError(404, "Not Found"); | |
| 571 | + return c.json(artifactJson(c, repo.id, a)); | |
| 572 | +}); | |
| 573 | + | |
| 574 | +actionsRest.get("/repos/:owner/:repo/actions/artifacts/:id/:format", async (c) => { | |
| 575 | + const { repo, stub } = await loadRepo(c, P(c).owner, P(c).repo); | |
| 576 | + if (c.req.param("format") !== "zip") throw new HttpError(404, "Not Found"); | |
| 577 | + const a = await stub.rtArtifactGet(idParam(c, "id"), { includeExpired: true }); | |
| 578 | + if (!a) throw new HttpError(404, "Not Found"); | |
| 579 | + // As on GitHub: an expired artifact is still listed, but its content is gone. | |
| 580 | + if (a.expired) throw new HttpError(410, "Artifact has expired"); | |
| 581 | + return c.redirect(await signBlobUrl(c.env.MASTER_KEY, origin(c.env), { repo: repo.id, kind: "artifact", id: a.id }, "r", nowS() + 600, blobName(a)), 302); | |
| 582 | +}); | |
| 583 | + | |
| 584 | +actionsRest.delete("/repos/:owner/:repo/actions/artifacts/:id", async (c) => { | |
| 585 | + const { repo, caller, stub } = await loadRepo(c, P(c).owner, P(c).repo, "write"); | |
| 586 | + const deleted = await stub.rtArtifactDelete(idParam(c, "id")); | |
| 587 | + if (!deleted) throw new HttpError(404, "Not Found"); | |
| 588 | + c.executionCtx.waitUntil(c.get("dir").audit({ actorId: caller.id, actorLogin: caller.login, onBehalfOf: caller.onBehalfOf, action: "actions.artifact.delete", target: `${repo.owner_login}/${repo.name}`, detail: { id: deleted.id, name: deleted.name } })); | |
| 589 | + return c.body(null, 204); | |
| 590 | +}); | |
| 591 | + | |
| 592 | +actionsRest.get("/repos/:owner/:repo/actions/caches", async (c) => { | |
| 593 | + const { stub } = await loadRepo(c, P(c).owner, P(c).repo); | |
| 594 | + const { perPage, page } = paging(c); | |
| 595 | + const res = await stub.rtListCaches({ key: c.req.query("key") || undefined, ref: c.req.query("ref") || undefined, sort: c.req.query("sort"), direction: c.req.query("direction"), page, perPage }); | |
| 596 | + return c.json({ total_count: res.total, actions_caches: res.caches.map(cacheJson) }); | |
| 597 | +}); | |
| 598 | + | |
| 599 | +actionsRest.delete("/repos/:owner/:repo/actions/caches", async (c) => { | |
| 600 | + const { repo, caller, stub } = await loadRepo(c, P(c).owner, P(c).repo, "write"); | |
| 601 | + const key = c.req.query("key"); | |
| 602 | + if (!key) throw new HttpError(422, "key is required"); | |
| 603 | + const deleted = await stub.rtDeleteCaches({ key, ref: c.req.query("ref") || undefined }); | |
| 604 | + if (!deleted.length) throw new HttpError(404, "Not Found"); | |
| 605 | + c.executionCtx.waitUntil(c.get("dir").audit({ actorId: caller.id, actorLogin: caller.login, onBehalfOf: caller.onBehalfOf, action: "actions.cache.delete", target: `${repo.owner_login}/${repo.name}`, detail: { key, ids: deleted.map((d) => d.id) } })); | |
| 606 | + return c.json({ total_count: deleted.length, actions_caches: deleted.map(cacheJson) }); | |
| 607 | +}); | |
| 608 | + | |
| 609 | +actionsRest.delete("/repos/:owner/:repo/actions/caches/:id", async (c) => { | |
| 610 | + const { repo, caller, stub } = await loadRepo(c, P(c).owner, P(c).repo, "write"); | |
| 611 | + const deleted = await stub.rtDeleteCaches({ id: idParam(c, "id") }); | |
| 612 | + if (!deleted.length) throw new HttpError(404, "Not Found"); | |
| 613 | + c.executionCtx.waitUntil(c.get("dir").audit({ actorId: caller.id, actorLogin: caller.login, onBehalfOf: caller.onBehalfOf, action: "actions.cache.delete", target: `${repo.owner_login}/${repo.name}`, detail: { key: deleted[0]!.key, ids: [deleted[0]!.id] } })); | |
| 614 | + return c.body(null, 204); | |
| 615 | +}); | |
| 616 | + | |
| 617 | +// The workflow token's defaults: GitHub's "Workflow permissions" settings. | |
| 618 | +const workflowSettingsJson = (s: { defaultPermissions: "read" | "write"; canApprovePullRequests: boolean }) => ({ default_workflow_permissions: s.defaultPermissions, can_approve_pull_request_reviews: s.canApprovePullRequests }); | |
| 619 | + | |
| 620 | +actionsRest.get("/repos/:owner/:repo/actions/permissions/workflow", async (c) => { | |
| 621 | + const { stub } = await loadRepo(c, P(c).owner, P(c).repo, "admin"); | |
| 622 | + return c.json(workflowSettingsJson(await stub.workflowSettings())); | |
| 623 | +}); | |
| 624 | + | |
| 625 | +actionsRest.put("/repos/:owner/:repo/actions/permissions/workflow", async (c) => { | |
| 626 | + const { caller, stub } = await loadRepo(c, P(c).owner, P(c).repo, "admin"); | |
| 627 | + const b = (await c.req.json().catch(() => null)) as { default_workflow_permissions?: unknown; can_approve_pull_request_reviews?: unknown } | null; | |
| 628 | + if (!b || typeof b !== "object") throw new HttpError(422, "a JSON body is required"); | |
| 629 | + // Audited by the repository (an audit event per change). | |
| 630 | + await stub.setWorkflowSettings({ defaultPermissions: b.default_workflow_permissions as "read" | "write" | undefined, canApprovePullRequests: b.can_approve_pull_request_reviews as boolean | undefined }, caller); | |
| 631 | + return c.body(null, 204); | |
| 632 | +}); | |
| 633 | + | |
| 634 | +actionsRest.get("/repos/:owner/:repo/actions/cache/usage", async (c) => { | |
| 635 | + const { stub } = await loadRepo(c, P(c).owner, P(c).repo); | |
| 636 | + const u = await stub.rtCacheUsage(); | |
| 637 | + return c.json({ full_name: `${P(c).owner}/${P(c).repo}`, active_caches_size_in_bytes: u.size, active_caches_count: u.count }); | |
| 638 | +}); |
modifiedapps/diffinery/src/http/api.ts+47−41View file
| @@ -1,7 +1,7 @@ | ||
| 1 | 1 | import { Hono } from "hono"; |
| 2 | 2 | import { fromUtf8 } from "@diffinery/git-core"; |
| 3 | 3 | import type { Env } from "../env.js"; |
| 4 | -import { parseScopes, requireAccount, requireAccountScope, type AppContext, type AppVars } from "../auth/principal.js"; | |
| 4 | +import { callerFor, parseScopes, requireAccount, requireAccountScope, type AppContext, type AppVars } from "../auth/principal.js"; | |
| 5 | 5 | import { baseRole, type Account, type Repo, type TokenScope } from "../lib/directory.js"; |
| 6 | 6 | import { encryptSecret } from "../lib/crypto.js"; |
| 7 | 7 | import { publicJwk } from "../lib/jwt.js"; |
| @@ -10,8 +10,9 @@ | ||
| 10 | 10 | import type { JobInfo, RunInfo } from "../repo/actions.js"; |
| 11 | 11 | import type { CommitInfo } from "../repo/RepoDO.js"; |
| 12 | 12 | import type { QueueEntry } from "../repo/queue.js"; |
| 13 | -import { loadRepo, refuseAgent, repoStub } from "./common.js"; | |
| 13 | +import { auditRepo, loadRepo, refuseAgent, repoStub } from "./common.js"; | |
| 14 | 14 | import { repoCreated } from "../mail/notify.js"; |
| 15 | +import { actionsRest } from "./actions-runtime.js"; | |
| 15 | 16 | import { compat } from "./compat.js"; |
| 16 | 17 | import { baseOf, encPath, ghCheckRun, ghComment, ghCommit, ghGitCommit, ghJob, ghRef, ghRun, ghFile, ghLabel, ghReview, ghReviewComment, ghUser, isoSeconds, nodeId, pageOf, paginate, repoApi, repoHtml, setLinks } from "./gh.js"; |
| 17 | 18 | import { decodePath, issueCtx, issuesOut, pullDetail, pullsOut, repoOut } from "./ghctx.js"; |
| @@ -173,15 +174,19 @@ | ||
| 173 | 174 | return c.json({ ...repoOut(c, repo, caller.role), open_issues_count: stats.openIssues + stats.openPulls, open_issues: stats.openIssues + stats.openPulls, stats }); |
| 174 | 175 | }); |
| 175 | 176 | api.patch("/repos/:owner/:repo", async (c) => { |
| 176 | - const { repo, caller } = await loadRepo(c, P(c).owner, P(c).repo, "admin"); | |
| 177 | + const { repo, caller } = await loadRepo(c, P(c).owner, P(c).repo, "admin", { whileArchived: true }); | |
| 177 | 178 | const b = await body<{ name?: string; description?: string; private?: boolean; visibility?: Repo["visibility"]; default_branch?: string; archived?: boolean }>(c); |
| 178 | 179 | if (b.private !== undefined && typeof b.private !== "boolean") throw new HttpError(422, "private must be a boolean", { errors: [{ resource: "Repository", code: "invalid", field: "private" }] }); |
| 179 | 180 | // GitHub's `private: false` makes the repository public. |
| 180 | 181 | const visibility = b.visibility ?? (b.private === true ? "private" : b.private === false ? "public" : undefined); |
| 181 | - return c.json(repoOut(c, await c.get("dir").updateRepo(repo, { name: b.name, description: b.description, visibility, defaultBranch: b.default_branch, archived: b.archived }), caller.role)); | |
| 182 | + const updated = await c.get("dir").updateRepo(repo, { name: b.name, description: b.description, visibility, defaultBranch: b.default_branch, archived: b.archived }); | |
| 183 | + const changed = Object.fromEntries(Object.entries({ name: b.name, description: b.description, visibility, default_branch: b.default_branch, archived: b.archived }).filter(([, v]) => v !== undefined)); | |
| 184 | + if (b.archived !== undefined && b.archived !== !!repo.archived) auditRepo(c, caller, repo, b.archived ? "repo.archive" : "repo.unarchive"); | |
| 185 | + if (Object.keys(changed).some((k) => k !== "archived")) auditRepo(c, caller, repo, "repo.update", changed); | |
| 186 | + return c.json(repoOut(c, updated, caller.role)); | |
| 182 | 187 | }); |
| 183 | 188 | api.delete("/repos/:owner/:repo", async (c) => { |
| 184 | - const { repo, caller } = await loadRepo(c, P(c).owner, P(c).repo, "admin"); | |
| 189 | + const { repo, caller } = await loadRepo(c, P(c).owner, P(c).repo, "admin", { whileArchived: true }); | |
| 185 | 190 | if (caller.kind === "agent") throw new HttpError(403, "agents cannot delete repositories"); |
| 186 | 191 | await c.get("dir").deleteRepo(repo); |
| 187 | 192 | c.executionCtx.waitUntil(c.get("dir").audit({ actorId: caller.id, actorLogin: caller.login, action: "repo.delete", target: `${repo.owner_login}/${repo.name}` })); |
| @@ -192,18 +197,23 @@ | ||
| 192 | 197 | return c.json(paginate(c, (await c.get("dir").collaborators(repo.id)).map((x) => ({ ...userJson(c, x.account), permissions: permissionsOf(x.role), role_name: x.role })))); |
| 193 | 198 | }); |
| 194 | 199 | api.put("/repos/:owner/:repo/collaborators/:login", async (c) => { |
| 195 | - const { repo } = await loadRepo(c, P(c).owner, P(c).repo, "admin"); | |
| 200 | + const { repo, caller } = await loadRepo(c, P(c).owner, P(c).repo, "admin"); | |
| 196 | 201 | const who = await c.get("dir").accountByLogin(c.req.param("login")); |
| 197 | 202 | if (!who) throw new HttpError(404, "user not found"); |
| 198 | 203 | const b = await body<{ permission?: Role | "pull" | "push" }>(c).catch(() => ({ permission: "write" as Role })); |
| 199 | 204 | const map: Record<string, Role> = { pull: "read", push: "write" }; |
| 200 | - await c.get("dir").setCollaborator(repo.id, who.id, map[b.permission ?? ""] ?? (b.permission as Role) ?? "write"); | |
| 205 | + const role = map[b.permission ?? ""] ?? (b.permission as Role) ?? "write"; | |
| 206 | + await c.get("dir").setCollaborator(repo.id, who.id, role); | |
| 207 | + auditRepo(c, caller, repo, "repo.collaborator.set", { login: who.login, role }); | |
| 201 | 208 | return c.body(null, 204); |
| 202 | 209 | }); |
| 203 | 210 | api.delete("/repos/:owner/:repo/collaborators/:login", async (c) => { |
| 204 | - const { repo } = await loadRepo(c, P(c).owner, P(c).repo, "admin"); | |
| 211 | + const { repo, caller } = await loadRepo(c, P(c).owner, P(c).repo, "admin"); | |
| 205 | 212 | const who = await c.get("dir").accountByLogin(c.req.param("login")); |
| 206 | - if (who) await c.get("dir").setCollaborator(repo.id, who.id, null); | |
| 213 | + if (who) { | |
| 214 | + await c.get("dir").setCollaborator(repo.id, who.id, null); | |
| 215 | + auditRepo(c, caller, repo, "repo.collaborator.remove", { login: who.login }); | |
| 216 | + } | |
| 207 | 217 | return c.body(null, 204); |
| 208 | 218 | }); |
| 209 | 219 | |
| @@ -372,10 +382,10 @@ | ||
| 372 | 382 | /** Agent-friendly multi-file commit without a Git client. */ |
| 373 | 383 | api.post("/repos/:owner/:repo/diffinery/commits", async (c) => { |
| 374 | 384 | const { stub, caller } = await loadRepo(c, P(c).owner, P(c).repo, "write"); |
| 375 | - const b = await body<{ branch: string; message: string; base?: string; expected_head_sha?: string; files: Array<{ path: string; content?: string; encoding?: "utf-8" | "base64"; delete?: boolean; mode?: "100644" | "100755" }> }>(c); | |
| 385 | + const b = await body<{ branch: string; message: string; base?: string; expected_head_sha?: string; allow_secrets?: string; files: Array<{ path: string; content?: string; encoding?: "utf-8" | "base64"; delete?: boolean; mode?: "100644" | "100755" }> }>(c); | |
| 376 | 386 | if (!Array.isArray(b.files) || b.files.length === 0) throw new HttpError(422, "files is required"); |
| 377 | 387 | const changes = b.files.map((f) => (f.delete ? { path: f.path, delete: true } : { path: f.path, mode: f.mode, content: f.encoding === "base64" ? fromB64(f.content ?? "") : new TextEncoder().encode(f.content ?? "") })); |
| 378 | - return c.json(await stub.commitFiles({ branch: b.branch, message: b.message, changes, createBranchFrom: b.base, expectedHeadSha: b.expected_head_sha }, caller), 201); | |
| 388 | + return c.json(await stub.commitFiles({ branch: b.branch, message: b.message, changes, createBranchFrom: b.base, expectedHeadSha: b.expected_head_sha, allowSecrets: typeof b.allow_secrets === "string" ? b.allow_secrets : undefined }, caller), 201); | |
| 379 | 389 | }); |
| 380 | 390 | |
| 381 | 391 | api.get("/repos/:owner/:repo/raw/:ref/*", async (c) => { |
| @@ -395,16 +405,18 @@ | ||
| 395 | 405 | return c.json(rule); |
| 396 | 406 | }); |
| 397 | 407 | api.put("/repos/:owner/:repo/branches/:branch/protection", async (c) => { |
| 398 | - const { stub, caller } = await loadRepo(c, P(c).owner, P(c).repo, "admin"); | |
| 408 | + const { stub, caller, repo } = await loadRepo(c, P(c).owner, P(c).repo, "admin"); | |
| 399 | 409 | if (caller.kind === "agent") throw new HttpError(403, "agents cannot change branch protection"); |
| 400 | 410 | const rules = await body<ProtectionRules>(c); |
| 401 | 411 | await stub.setProtection(c.req.param("branch"), rules); |
| 412 | + auditRepo(c, caller, repo, "repo.protection.set", { pattern: c.req.param("branch") }); | |
| 402 | 413 | return c.json({ pattern: c.req.param("branch"), rules }); |
| 403 | 414 | }); |
| 404 | 415 | api.delete("/repos/:owner/:repo/branches/:branch/protection", async (c) => { |
| 405 | - const { stub, caller } = await loadRepo(c, P(c).owner, P(c).repo, "admin"); | |
| 416 | + const { stub, caller, repo } = await loadRepo(c, P(c).owner, P(c).repo, "admin"); | |
| 406 | 417 | if (caller.kind === "agent") throw new HttpError(403, "agents cannot change branch protection"); |
| 407 | 418 | await stub.setProtection(c.req.param("branch"), null); |
| 419 | + auditRepo(c, caller, repo, "repo.protection.delete", { pattern: c.req.param("branch") }); | |
| 408 | 420 | return c.body(null, 204); |
| 409 | 421 | }); |
| 410 | 422 | |
| @@ -702,41 +714,27 @@ | ||
| 702 | 714 | if (c.req.query("format") === "json") return c.json(lines); |
| 703 | 715 | return c.text(lines.map((l) => `${l.t} ${l.text}`).join("\n")); |
| 704 | 716 | }); |
| 705 | -api.get("/repos/:owner/:repo/actions/runs/:id/artifacts", async (c) => { | |
| 706 | - const { stub } = await loadRepo(c, P(c).owner, P(c).repo); | |
| 707 | - const list = await stub.artifacts(c.req.param("id")); | |
| 708 | - const runId = c.req.param("id"); | |
| 709 | - return c.json({ | |
| 710 | - total_count: list.length, | |
| 711 | - artifacts: paginate(c, list).map((a) => ({ | |
| 712 | - name: a.name, | |
| 713 | - size_in_bytes: a.size, | |
| 714 | - archive_download_url: `${c.env.PUBLIC_URL}/${P(c).owner}/${P(c).repo}/actions/runs/${runId}/artifacts/${encodeURIComponent(a.name)}`, | |
| 715 | - expired: false, | |
| 716 | - created_at: isoSeconds(a.createdAt), | |
| 717 | - updated_at: isoSeconds(a.createdAt), | |
| 718 | - workflow_run: { id: runId }, | |
| 719 | - })), | |
| 720 | - }); | |
| 721 | -}); | |
| 717 | +api.route("/", actionsRest); | |
| 722 | 718 | api.get("/repos/:owner/:repo/actions/secrets", async (c) => { |
| 723 | 719 | const { stub } = await loadRepo(c, P(c).owner, P(c).repo, "admin"); |
| 724 | 720 | const s = await stub.secretNames(); |
| 725 | 721 | return c.json({ total_count: s.length, secrets: s.map((x) => ({ name: x.name, updated_at: x.updatedAt })) }); |
| 726 | 722 | }); |
| 727 | 723 | api.put("/repos/:owner/:repo/actions/secrets/:name", async (c) => { |
| 728 | - const { stub, caller } = await loadRepo(c, P(c).owner, P(c).repo, "admin"); | |
| 724 | + const { stub, caller, repo } = await loadRepo(c, P(c).owner, P(c).repo, "admin"); | |
| 729 | 725 | if (caller.kind === "agent") throw new HttpError(403, "agents cannot manage secrets"); |
| 730 | 726 | const b = await body<{ value: string; encrypted_value?: string }>(c); |
| 731 | 727 | if (typeof b.value !== "string" && typeof b.encrypted_value === "string") throw new HttpError(422, "encrypted_value (sealed box) is not supported yet: send the secret as `value` over HTTPS; it is encrypted at rest"); |
| 732 | 728 | if (typeof b.value !== "string") throw new HttpError(422, "value is required (send it over HTTPS; it is encrypted at rest)"); |
| 733 | 729 | await stub.setSecret(c.req.param("name"), b.value); |
| 730 | + auditRepo(c, caller, repo, "repo.secret.set", { name: c.req.param("name").toUpperCase() }); | |
| 734 | 731 | return c.body(null, 204); |
| 735 | 732 | }); |
| 736 | 733 | api.delete("/repos/:owner/:repo/actions/secrets/:name", async (c) => { |
| 737 | - const { stub, caller } = await loadRepo(c, P(c).owner, P(c).repo, "admin"); | |
| 734 | + const { stub, caller, repo } = await loadRepo(c, P(c).owner, P(c).repo, "admin"); | |
| 738 | 735 | refuseAgent(caller, "secrets"); |
| 739 | 736 | await stub.deleteSecret(c.req.param("name")); |
| 737 | + auditRepo(c, caller, repo, "repo.secret.delete", { name: c.req.param("name").toUpperCase() }); | |
| 740 | 738 | return c.body(null, 204); |
| 741 | 739 | }); |
| 742 | 740 | api.get("/repos/:owner/:repo/actions/variables", async (c) => { |
| @@ -745,15 +743,17 @@ | ||
| 745 | 743 | return c.json({ total_count: all.length, variables: paginate(c, all) }); |
| 746 | 744 | }); |
| 747 | 745 | api.delete("/repos/:owner/:repo/actions/variables/:name", async (c) => { |
| 748 | - const { stub, caller } = await loadRepo(c, P(c).owner, P(c).repo, "admin"); | |
| 746 | + const { stub, caller, repo } = await loadRepo(c, P(c).owner, P(c).repo, "admin"); | |
| 749 | 747 | refuseAgent(caller, "variables"); |
| 750 | 748 | await stub.setVariable(c.req.param("name"), null); |
| 749 | + auditRepo(c, caller, repo, "repo.variable.delete", { name: c.req.param("name").toUpperCase() }); | |
| 751 | 750 | return c.body(null, 204); |
| 752 | 751 | }); |
| 753 | 752 | api.put("/repos/:owner/:repo/actions/variables/:name", async (c) => { |
| 754 | - const { stub, caller } = await loadRepo(c, P(c).owner, P(c).repo, "admin"); | |
| 753 | + const { stub, caller, repo } = await loadRepo(c, P(c).owner, P(c).repo, "admin"); | |
| 755 | 754 | refuseAgent(caller, "variables"); |
| 756 | 755 | await stub.setVariable(c.req.param("name"), (await body<{ value: string }>(c)).value); |
| 756 | + auditRepo(c, caller, repo, "repo.variable.set", { name: c.req.param("name").toUpperCase() }); | |
| 757 | 757 | return c.body(null, 204); |
| 758 | 758 | }); |
| 759 | 759 | async function orgAdmin(c: AppContext) { |
| @@ -773,16 +773,18 @@ | ||
| 773 | 773 | return c.json({ total_count: results.length, secrets: results }); |
| 774 | 774 | }); |
| 775 | 775 | api.delete("/orgs/:org/actions/secrets/:name", async (c) => { |
| 776 | - const { org } = await orgAdmin(c); | |
| 776 | + const { a, org } = await orgAdmin(c); | |
| 777 | 777 | await c.env.DB.prepare("DELETE FROM org_secrets WHERE org_id = ? AND name = ?").bind(org.id, c.req.param("name").toUpperCase()).run(); |
| 778 | + c.executionCtx.waitUntil(c.get("dir").audit({ actorId: a.id, actorLogin: a.login, action: "org.secret.delete", target: org.login, detail: { name: c.req.param("name").toUpperCase(), via: "api" } })); | |
| 778 | 779 | return c.body(null, 204); |
| 779 | 780 | }); |
| 780 | 781 | api.put("/orgs/:org/actions/secrets/:name", async (c) => { |
| 781 | - const { org } = await orgAdmin(c); | |
| 782 | + const { a, org } = await orgAdmin(c); | |
| 782 | 783 | const name = c.req.param("name").toUpperCase(); |
| 783 | 784 | if (!/^[A-Z_][A-Z0-9_]*$/.test(name)) throw new HttpError(422, "invalid secret name"); |
| 784 | 785 | const { value } = await body<{ value: string }>(c); |
| 785 | 786 | await c.env.DB.prepare("INSERT OR REPLACE INTO org_secrets (org_id, name, ciphertext, updated_at) VALUES (?, ?, ?, ?)").bind(org.id, name, await encryptSecret(c.env.MASTER_KEY, `org:${org.id}`, value), new Date().toISOString()).run(); |
| 787 | + c.executionCtx.waitUntil(c.get("dir").audit({ actorId: a.id, actorLogin: a.login, action: "org.secret.set", target: org.login, detail: { name, via: "api" } })); | |
| 786 | 788 | return c.body(null, 204); |
| 787 | 789 | }); |
| 788 | 790 | |
| @@ -793,17 +795,19 @@ | ||
| 793 | 795 | return c.json(paginate(c, (await stub.webhooks()).map(({ secret: _s, ...h }) => hookJson(c, h)))); |
| 794 | 796 | }); |
| 795 | 797 | api.post("/repos/:owner/:repo/hooks", async (c) => { |
| 796 | - const { stub } = await loadRepo(c, P(c).owner, P(c).repo, "admin"); | |
| 798 | + const { stub, caller, repo } = await loadRepo(c, P(c).owner, P(c).repo, "admin"); | |
| 797 | 799 | const b = await body<{ config: { url: string; secret?: string }; events?: string[] }>(c); |
| 798 | 800 | if (typeof b.config?.url !== "string") throw new HttpError(422, "config.url is required", { errors: [{ resource: "Hook", code: "missing_field", field: "config.url" }] }); |
| 799 | 801 | const created = await stub.createWebhook(b.config.url, Array.isArray(b.events) ? b.events.map(String) : ["*"], b.config.secret); |
| 800 | 802 | const hook = (await stub.webhooks()).find((h) => h.id === created.id)!; |
| 803 | + auditRepo(c, caller, repo, "repo.webhook.create", { id: created.id, url: b.config.url }); | |
| 801 | 804 | // The generated secret is shown once, here (GitHub never returns secrets; Diffinery makes one when none is given). |
| 802 | 805 | return c.json({ ...hookJson(c, { ...hook, secret: undefined }), secret: b.config.secret ? undefined : created.secret }, 201); |
| 803 | 806 | }); |
| 804 | 807 | api.delete("/repos/:owner/:repo/hooks/:id", async (c) => { |
| 805 | - const { stub } = await loadRepo(c, P(c).owner, P(c).repo, "admin"); | |
| 808 | + const { stub, caller, repo } = await loadRepo(c, P(c).owner, P(c).repo, "admin"); | |
| 806 | 809 | await stub.deleteWebhook(c.req.param("id")); |
| 810 | + auditRepo(c, caller, repo, "repo.webhook.delete", { id: c.req.param("id") }); | |
| 807 | 811 | return c.body(null, 204); |
| 808 | 812 | }); |
| 809 | 813 | |
| @@ -811,9 +815,10 @@ | ||
| 811 | 815 | |
| 812 | 816 | api.get("/diffinery/tokens", async (c) => c.json(await c.get("dir").tokens(requireAccount(c).id))); |
| 813 | 817 | api.post("/diffinery/tokens", async (c) => { |
| 818 | + // Minting a token needs a browser session or an admin token for the whole account: a token limited | |
| 819 | + // to some repositories must not be able to create one that isn't. | |
| 820 | + requireAccountScope(c, "admin"); | |
| 814 | 821 | const a = requireAccount(c); |
| 815 | - const p = c.get("principal"); | |
| 816 | - if (p.type === "account" && p.via === "token" && !p.scopes.includes("admin")) throw new HttpError(403, "creating tokens requires an admin-scoped token or a browser session"); | |
| 817 | 822 | const b = await body<{ name: string; scopes?: TokenScope[]; repositories?: string[]; expires_in_days?: number }>(c); |
| 818 | 823 | const scopes = parseScopes(b.scopes); |
| 819 | 824 | let repoIds: string[] | null = null; |
| @@ -822,7 +827,8 @@ | ||
| 822 | 827 | for (const full of b.repositories) { |
| 823 | 828 | const [o, n] = full.split("/"); |
| 824 | 829 | const r = o && n ? await c.get("dir").repo(o, n) : null; |
| 825 | - if (!r) throw new HttpError(422, `unknown repository ${full}`); | |
| 830 | + // Repositories the account can't see are "unknown", so this doesn't reveal private ones. | |
| 831 | + if (!r || !(await callerFor(c, r))) throw new HttpError(422, `unknown repository ${full}`); | |
| 826 | 832 | repoIds.push(r.id); |
| 827 | 833 | } |
| 828 | 834 | } |
modifiedapps/diffinery/src/http/common.ts+25−3View file
| @@ -1,5 +1,6 @@ | ||
| 1 | 1 | import type { Env } from "../env.js"; |
| 2 | 2 | import type { Repo } from "../lib/directory.js"; |
| 3 | +import { denied, readOnly, requestPermission, type JobPermissions } from "../lib/job-permissions.js"; | |
| 3 | 4 | import { HttpError } from "../lib/util.js"; |
| 4 | 5 | import { GITHUB_DOCS } from "./gh.js"; |
| 5 | 6 | import { callerFor, requireScope, type AppContext } from "../auth/principal.js"; |
| @@ -13,24 +14,45 @@ | ||
| 13 | 14 | * may only read: any other method needs a signed-in caller, as on GitHub, |
| 14 | 15 | * unless the route is a read that happens to use POST (git fetch, LFS |
| 15 | 16 | * download) and says so. Writes also need a token with the write scope. |
| 17 | + * Archived repositories refuse every write, whatever role the route needs, | |
| 18 | + * except the few routes that pass `whileArchived` (unarchiving, deleting, | |
| 19 | + * watching), as on GitHub. | |
| 16 | 20 | */ |
| 17 | -export async function loadRepo(c: AppContext, owner: string, name: string, min: Role = "read", opts: { readOnlyPost?: boolean } = {}): Promise<{ repo: Repo; caller: Caller; stub: ReturnType<typeof repoStub> }> { | |
| 21 | +export async function loadRepo(c: AppContext, owner: string, name: string, min: Role = "read", opts: { readOnlyPost?: boolean; whileArchived?: boolean } = {}): Promise<{ repo: Repo; caller: Caller; stub: ReturnType<typeof repoStub> }> { | |
| 18 | 22 | const repo = await c.get("dir").repo(owner, name.replace(/\.git$/, "")); |
| 19 | 23 | if (!repo) throw new HttpError(404, "repository not found"); |
| 20 | 24 | const anonymous = c.get("principal").type === "anonymous"; |
| 21 | 25 | const reading = c.req.method === "GET" || c.req.method === "HEAD" || (opts.readOnlyPost && c.req.method === "POST") || c.get("readOnly") === true; |
| 22 | 26 | if (anonymous && !reading) throw new HttpError(401, "authentication required"); |
| 23 | - if (!reading) requireScope(c, "write"); | |
| 24 | 27 | const caller = await callerFor(c, repo); |
| 25 | 28 | if (!caller) { |
| 26 | 29 | if (anonymous) throw new HttpError(401, "authentication required"); |
| 27 | 30 | throw new HttpError(404, "repository not found"); |
| 28 | 31 | } |
| 32 | + // After the visibility check, so a read token's write to a private repository it can't see is still a 404. | |
| 33 | + if (!reading) requireScope(c, "write"); | |
| 29 | 34 | if (!atLeast(caller.role, min)) throw new HttpError(403, `${min} access required`); |
| 30 | - if (repo.archived && min !== "read" && min !== "admin") throw new HttpError(403, "repository is archived (read-only)"); | |
| 35 | + if (caller.permissions) { | |
| 36 | + // A workflow's GITHUB_TOKEN: each kind of endpoint needs its permission (issues, contents, ...). | |
| 37 | + // What anyone may read in a public repository stays readable with `permissions: { contents: none }`. | |
| 38 | + const req = requestPermission(c.req.method, c.req.path, min); | |
| 39 | + const reason = denied(caller.permissions as JobPermissions, req); | |
| 40 | + if (reason && !(repo.visibility === "public" && readOnly(req))) throw new HttpError(403, reason); | |
| 41 | + } | |
| 42 | + if (repo.archived && !reading && !opts.whileArchived) throw new HttpError(403, "repository is archived (read-only)"); | |
| 31 | 43 | return { repo, caller, stub: repoStub(c.env, repo.id) }; |
| 32 | 44 | } |
| 33 | 45 | |
| 46 | +/** Record a repository settings change made through the API, like the web settings pages do. */ | |
| 47 | +export function auditRepo(c: AppContext, caller: Caller, repo: Pick<Repo, "owner_login" | "name">, action: string, detail: Record<string, unknown> = {}): void { | |
| 48 | + c.executionCtx.waitUntil( | |
| 49 | + c | |
| 50 | + .get("dir") | |
| 51 | + .audit({ actorId: caller.id, actorLogin: caller.login, onBehalfOf: caller.onBehalfOf, action, target: `${repo.owner_login}/${repo.name}`, detail: { ...detail, via: "api" } }) | |
| 52 | + .catch((e) => console.error("audit failed", e)), | |
| 53 | + ); | |
| 54 | +} | |
| 55 | + | |
| 34 | 56 | /** Secrets and Actions variables are managed by people only. */ |
| 35 | 57 | export function refuseAgent(caller: Caller, what = "secrets and variables"): void { |
| 36 | 58 | if (caller.kind === "agent") throw new HttpError(403, `agents cannot manage ${what}`); |
modifiedapps/diffinery/src/http/deployments.ts+8−2View file
| @@ -68,7 +68,8 @@ | ||
| 68 | 68 | node_id: nodeId("Environment", `${repoParams(c).owner}/${repoParams(c).repo}/${e.id}`), |
| 69 | 69 | name: e.name, |
| 70 | 70 | url: `${api(c)}/environments/${name}`, |
| 71 | - html_url: `${repoHtml(baseOf(c), repoParams(c).owner, repoParams(c).repo)}/deployments/activity_log?environments_filter=${name}`, | |
| 71 | + // Diffinery has no environments page yet; deployments show on the Actions runs that made them. | |
| 72 | + html_url: `${repoHtml(baseOf(c), repoParams(c).owner, repoParams(c).repo)}/actions`, | |
| 72 | 73 | created_at: isoSeconds(e.createdAt), |
| 73 | 74 | updated_at: isoSeconds(e.updatedAt), |
| 74 | 75 | can_admins_bypass: true, |
| @@ -221,6 +222,7 @@ | ||
| 221 | 222 | if (typeof b.name !== "string" || typeof b.value !== "string") throw new HttpError(422, "name and value are required"); |
| 222 | 223 | if ((await stub.environmentVariables(envName(c))).some((x) => x.name === b.name!.toUpperCase())) throw new HttpError(409, "Already exists - Variable already exists"); |
| 223 | 224 | await stub.setEnvironmentVariable(envName(c), b.name, b.value, caller); |
| 225 | + audit(c, caller, "environment.variable.set", { environment: envName(c), name: b.name.toUpperCase() }); | |
| 224 | 226 | return c.json({}, 201); |
| 225 | 227 | }); |
| 226 | 228 | deployments.patch(`${R}/environments/:env/variables/:name`, async (c) => { |
| @@ -228,16 +230,20 @@ | ||
| 228 | 230 | refuseAgent(caller, "variables"); |
| 229 | 231 | const b = await jsonBody<{ name?: string; value?: string }>(c); |
| 230 | 232 | const name = c.req.param("name"); |
| 231 | - const current = (await stub.environmentVariables(envName(c))).find((x) => x.name === name.toUpperCase()); | |
| 233 | + const all = await stub.environmentVariables(envName(c)); | |
| 234 | + const current = all.find((x) => x.name === name.toUpperCase()); | |
| 232 | 235 | if (!current) throw new HttpError(404, "Not Found"); |
| 233 | 236 | const next = typeof b.name === "string" && b.name ? b.name : name; |
| 237 | + if (next.toUpperCase() !== name.toUpperCase() && all.some((x) => x.name === next.toUpperCase())) throw new HttpError(409, "Already exists - Variable already exists"); | |
| 234 | 238 | await stub.setEnvironmentVariable(envName(c), next, typeof b.value === "string" ? b.value : current.value, caller); |
| 235 | 239 | if (next.toUpperCase() !== name.toUpperCase()) await stub.setEnvironmentVariable(envName(c), name, null, caller); |
| 240 | + audit(c, caller, "environment.variable.set", { environment: envName(c), name: next.toUpperCase(), ...(next.toUpperCase() !== name.toUpperCase() ? { renamed_from: name.toUpperCase() } : {}) }); | |
| 236 | 241 | return c.body(null, 204); |
| 237 | 242 | }); |
| 238 | 243 | deployments.delete(`${R}/environments/:env/variables/:name`, async (c) => { |
| 239 | 244 | const { stub, caller } = await load(c, "admin"); |
| 240 | 245 | refuseAgent(caller, "variables"); |
| 241 | 246 | await stub.setEnvironmentVariable(envName(c), c.req.param("name"), null, caller); |
| 247 | + audit(c, caller, "environment.variable.delete", { environment: envName(c), name: c.req.param("name").toUpperCase() }); | |
| 242 | 248 | return c.body(null, 204); |
| 243 | 249 | }); |
modifiedapps/diffinery/src/http/git.ts+4−1View file
| @@ -15,7 +15,10 @@ | ||
| 15 | 15 | git.get("/:owner/:repo/info/refs", async (c) => { |
| 16 | 16 | const service = c.req.query("service"); |
| 17 | 17 | if (service !== "git-upload-pack" && service !== "git-receive-pack") throw new HttpError(403, "dumb HTTP protocol is not supported; update git"); |
| 18 | - const { stub } = await loadRepo(c, c.req.param("owner"), c.req.param("repo"), service === "git-receive-pack" ? "write" : "read"); | |
| 18 | + // Git only offers credentials after a 401, so an anonymous push must be asked for them (as GitHub does), even to a public repository. | |
| 19 | + if (service === "git-receive-pack" && c.get("principal").type === "anonymous") throw new HttpError(401, "authentication required"); | |
| 20 | + const { stub, repo } = await loadRepo(c, c.req.param("owner"), c.req.param("repo"), service === "git-receive-pack" ? "write" : "read"); | |
| 21 | + if (service === "git-receive-pack" && repo.archived) throw new HttpError(403, "repository is archived (read-only)"); | |
| 19 | 22 | if (service === "git-upload-pack" && !/version=2/.test(c.req.header("git-protocol") ?? "")) { |
| 20 | 23 | throw new HttpError(400, "Diffinery requires Git protocol v2 (git 2.26 or newer, or run: git config --global protocol.version 2)"); |
| 21 | 24 | } |
modifiedapps/diffinery/src/http/misc.ts+6−3View file
| @@ -4,7 +4,7 @@ | ||
| 4 | 4 | import { HttpError } from "../lib/util.js"; |
| 5 | 5 | import { baseOf, ghRun, isoSeconds, nodeId, repoApi, repoHtml } from "./gh.js"; |
| 6 | 6 | import { intParam, jsonBody, loadTarget, repoParams } from "./ghctx.js"; |
| 7 | -import { refuseAgent } from "./common.js"; | |
| 7 | +import { auditRepo, refuseAgent } from "./common.js"; | |
| 8 | 8 | |
| 9 | 9 | /** |
| 10 | 10 | * Smaller GitHub endpoints scripts and actions call: rate limits, topics, |
| @@ -126,24 +126,27 @@ | ||
| 126 | 126 | return c.json({ name, value: vars[name], created_at: null, updated_at: null }); |
| 127 | 127 | }); |
| 128 | 128 | misc.post(`${R}/actions/variables`, async (c) => { |
| 129 | - const { stub, caller } = await load(c, "admin"); | |
| 129 | + const { stub, caller, repo } = await load(c, "admin"); | |
| 130 | 130 | refuseAgent(caller, "variables"); |
| 131 | 131 | const b = await jsonBody<{ name?: string; value?: string }>(c); |
| 132 | 132 | if (typeof b.name !== "string" || typeof b.value !== "string") throw new HttpError(422, "name and value are required", { errors: [{ resource: "ActionsVariable", code: "missing_field", field: typeof b.name !== "string" ? "name" : "value" }] }); |
| 133 | 133 | if (b.name.toUpperCase() in (await stub.variables())) throw new HttpError(409, "Already exists - Variable already exists"); |
| 134 | 134 | await stub.setVariable(b.name, b.value); |
| 135 | + auditRepo(c, caller, repo, "repo.variable.set", { name: b.name.toUpperCase() }); | |
| 135 | 136 | return c.json({}, 201); |
| 136 | 137 | }); |
| 137 | 138 | misc.patch(`${R}/actions/variables/:name`, async (c) => { |
| 138 | - const { stub, caller } = await load(c, "admin"); | |
| 139 | + const { stub, caller, repo } = await load(c, "admin"); | |
| 139 | 140 | refuseAgent(caller, "variables"); |
| 140 | 141 | const name = c.req.param("name"); |
| 141 | 142 | const vars = await stub.variables(); |
| 142 | 143 | if (!(name.toUpperCase() in vars)) throw new HttpError(404, "Not Found"); |
| 143 | 144 | const b = await jsonBody<{ name?: string; value?: string }>(c); |
| 144 | 145 | const next = typeof b.name === "string" && b.name ? b.name : name; |
| 146 | + if (next.toUpperCase() !== name.toUpperCase() && next.toUpperCase() in vars) throw new HttpError(409, "Already exists - Variable already exists"); | |
| 145 | 147 | await stub.setVariable(next, typeof b.value === "string" ? b.value : vars[name.toUpperCase()]!); |
| 146 | 148 | if (next.toUpperCase() !== name.toUpperCase()) await stub.setVariable(name.toUpperCase(), null); |
| 149 | + auditRepo(c, caller, repo, "repo.variable.set", { name: next.toUpperCase(), ...(next.toUpperCase() !== name.toUpperCase() ? { renamed_from: name.toUpperCase() } : {}) }); | |
| 147 | 150 | return c.body(null, 204); |
| 148 | 151 | }); |
| 149 | 152 |
modifiedapps/diffinery/src/http/runner.ts+37−12View file
| @@ -4,7 +4,10 @@ | ||
| 4 | 4 | import type { AppContext, AppVars } from "../auth/principal.js"; |
| 5 | 5 | import { credentialFrom } from "../auth/principal.js"; |
| 6 | 6 | import { signEs256 } from "../lib/jwt.js"; |
| 7 | +import { BodyTooLarge, putStream, sized, type BlobBucket } from "../lib/blob-store.js"; | |
| 8 | +import { getFirst, headFirst } from "../lib/storage-keys.js"; | |
| 7 | 9 | import { HttpError } from "../lib/util.js"; |
| 10 | +import { CACHE_LIMIT_BYTES } from "../repo/runtime.js"; | |
| 8 | 11 | import { repoStub } from "./common.js"; |
| 9 | 12 | |
| 10 | 13 | /** Endpoints the runner agent inside a job container calls. Auth: the job token. */ |
| @@ -18,6 +21,24 @@ | ||
| 18 | 21 | return { token, repoId }; |
| 19 | 22 | } |
| 20 | 23 | |
| 24 | +/** Stores a request body as it streams in, refusing more than `max` bytes (older runners' cache and artifact uploads). */ | |
| 25 | +async function storeBody(c: AppContext, key: string, max: number): Promise<number> { | |
| 26 | + const body = c.req.raw.body; | |
| 27 | + if (!body) throw new HttpError(400, "empty body"); | |
| 28 | + const options = { httpMetadata: { contentType: "application/gzip" } }; | |
| 29 | + const len = c.req.header("content-length"); | |
| 30 | + if (len !== undefined && /^\d{1,15}$/.test(len)) { | |
| 31 | + if (Number(len) > max) throw new HttpError(413, `uploads are limited to ${max} bytes`); | |
| 32 | + return (await c.env.BUCKET.put(key, sized(body, Number(len)), options)).size; | |
| 33 | + } | |
| 34 | + try { | |
| 35 | + return await putStream(c.env.BUCKET as unknown as BlobBucket, key, body, max, options); | |
| 36 | + } catch (e) { | |
| 37 | + if (e instanceof BodyTooLarge) throw new HttpError(413, `uploads are limited to ${max} bytes`); | |
| 38 | + throw e; | |
| 39 | + } | |
| 40 | +} | |
| 41 | + | |
| 21 | 42 | runner.get("/runner/jobs/:jobId", async (c) => { |
| 22 | 43 | const jobId = c.req.param("jobId"); |
| 23 | 44 | const { token, repoId } = jobAuth(c.req.raw, jobId); |
| @@ -55,14 +76,15 @@ | ||
| 55 | 76 | const restore = (c.req.query("restore") ?? "").split(",").map((s) => s.trim()).filter(Boolean); |
| 56 | 77 | const hit = await repoStub(c.env, repoId).cacheLookup(jobId, token, key, restore); |
| 57 | 78 | if (!hit) return c.json({ message: "cache miss" }, 404); |
| 58 | - return c.json({ key: hit.key, url: `${new URL(c.req.url).origin}/runner/jobs/${jobId}/cache/blob/${encodeURIComponent(hit.key)}` }); | |
| 79 | + return c.json({ key: hit.key, url: `${new URL(c.req.url).origin}/runner/jobs/${jobId}/cache/blob/${encodeURIComponent(hit.stored)}` }); | |
| 59 | 80 | }); |
| 60 | 81 | |
| 61 | 82 | runner.get("/runner/jobs/:jobId/cache/blob/:key", async (c) => { |
| 62 | 83 | const jobId = c.req.param("jobId"); |
| 63 | - const { repoId } = jobAuth(c.req.raw, jobId); | |
| 64 | - const r2Key = await repoStub(c.env, repoId).cacheKeyFor(c.req.param("key")); | |
| 65 | - const obj = await c.env.BUCKET.get(r2Key); | |
| 84 | + const { token, repoId } = jobAuth(c.req.raw, jobId); | |
| 85 | + // Only entries of a scope the job may restore from (its ref, the PR base, the default branch). | |
| 86 | + const keys = await repoStub(c.env, repoId).cacheReadKeys(jobId, token, c.req.param("key")); | |
| 87 | + const obj = keys && (await getFirst(c.env.BUCKET, keys)); | |
| 66 | 88 | if (!obj) throw new HttpError(404, "cache entry missing"); |
| 67 | 89 | return new Response(obj.body, { headers: { "content-type": "application/gzip", "content-length": String(obj.size) } }); |
| 68 | 90 | }); |
| @@ -73,10 +95,10 @@ | ||
| 73 | 95 | const key = c.req.param("key"); |
| 74 | 96 | if (key.length > 512) throw new HttpError(422, "cache key too long"); |
| 75 | 97 | const stub = repoStub(c.env, repoId); |
| 76 | - const r2Key = await stub.cacheKeyFor(key); | |
| 77 | - if (await c.env.BUCKET.head(r2Key)) return c.json({ message: "cache entry already exists" }, 409); | |
| 78 | - const put = await c.env.BUCKET.put(r2Key, c.req.raw.body!, { httpMetadata: { contentType: "application/gzip" } }); | |
| 79 | - const evicted = await stub.cacheSaved(jobId, token, key, put?.size ?? 0); | |
| 98 | + const { r2Keys: keys } = await stub.cacheSaveKeys(jobId, token, key); | |
| 99 | + if (await headFirst(c.env.BUCKET, keys)) return c.json({ message: "cache entry already exists" }, 409); | |
| 100 | + const size = await storeBody(c, keys[0]!, CACHE_LIMIT_BYTES); | |
| 101 | + const evicted = await stub.cacheSaved(jobId, token, key, size); | |
| 80 | 102 | if (evicted.length) await c.env.BUCKET.delete(evicted); |
| 81 | 103 | return c.json({ ok: true }); |
| 82 | 104 | }); |
| @@ -87,16 +109,19 @@ | ||
| 87 | 109 | const jobId = c.req.param("jobId"); |
| 88 | 110 | const { token, repoId } = jobAuth(c.req.raw, jobId); |
| 89 | 111 | const stub = repoStub(c.env, repoId); |
| 90 | - const r2Key = await stub.artifactKeyFor(jobId, c.req.param("name")); | |
| 91 | - const put = await c.env.BUCKET.put(r2Key, c.req.raw.body!, { httpMetadata: { contentType: "application/gzip" } }); | |
| 92 | - await stub.artifactSaved(jobId, token, c.req.param("name"), put?.size ?? 0); | |
| 112 | + const keys = await stub.artifactKeysFor(jobId, c.req.param("name")); | |
| 113 | + // What is left of the repository's artifact quota, counting live artifacts from both upload paths. | |
| 114 | + const room = await stub.artifactRoom(); | |
| 115 | + if (room === 0) throw new HttpError(413, "the repository's artifact storage quota is used up; delete artifacts or wait for them to expire"); | |
| 116 | + const size = await storeBody(c, keys[0]!, room); | |
| 117 | + await stub.artifactSaved(jobId, token, c.req.param("name"), size); | |
| 93 | 118 | return c.json({ ok: true }); |
| 94 | 119 | }); |
| 95 | 120 | |
| 96 | 121 | runner.get("/runner/jobs/:jobId/artifacts/:name", async (c) => { |
| 97 | 122 | const jobId = c.req.param("jobId"); |
| 98 | 123 | const { repoId } = jobAuth(c.req.raw, jobId); |
| 99 | - const obj = await c.env.BUCKET.get(await repoStub(c.env, repoId).artifactKeyFor(jobId, c.req.param("name"))); | |
| 124 | + const obj = await getFirst(c.env.BUCKET, await repoStub(c.env, repoId).artifactKeysFor(jobId, c.req.param("name"))); | |
| 100 | 125 | if (!obj) throw new HttpError(404, "artifact not found"); |
| 101 | 126 | return new Response(obj.body, { headers: { "content-type": "application/gzip" } }); |
| 102 | 127 | }); |
addedapps/diffinery/src/lib/blob-store.ts+391−0View file
| @@ -1,0 +1,391 @@ | ||
| 1 | +import type { BlobRef } from "./runtime-auth.js"; | |
| 2 | +import { STAGING_ROOT, runtimeObjectKey } from "./storage-keys.js"; | |
| 3 | +import { b64url } from "./util.js"; | |
| 4 | + | |
| 5 | +/** | |
| 6 | + * R2 layout for Actions cache entries and artifacts, and the upload protocol the | |
| 7 | + * toolkit uses: Azure Blob "Put Block" + "Put Block List" (or one "Put Blob"), | |
| 8 | + * and the legacy cache API's ranged PATCH chunks. Blocks and chunks are staged | |
| 9 | + * as separate objects and, on commit, composed into one object, so downloads are | |
| 10 | + * a single (rangeable) R2 read. | |
| 11 | + * | |
| 12 | + * Staged objects live under `actions-staging/`; an R2 lifecycle rule on that | |
| 13 | + * prefix removes uploads that were never committed (lib/storage-keys.ts). | |
| 14 | + */ | |
| 15 | + | |
| 16 | +export { STAGING_ROOT }; | |
| 17 | + | |
| 18 | +/** Smallest multipart part R2 accepts (except the last); R2 also wants every other part the same size. */ | |
| 19 | +const MIN_PART = 5 * 1024 * 1024; | |
| 20 | +const MAX_PARTS = 10_000; | |
| 21 | +const MiB = 1024 * 1024; | |
| 22 | + | |
| 23 | +/** | |
| 24 | + * R2 reads (and writes) one composition step makes. A Worker invocation may make | |
| 25 | + * 1000 subrequests and binding calls count, so a step stays well under that and | |
| 26 | + * each step runs in an invocation of its own (a Durable Object call). | |
| 27 | + */ | |
| 28 | +export const STEP_OPS = 400; | |
| 29 | +/** Largest object a merge step writes (an intermediate, or a final object written in one put). */ | |
| 30 | +const MAX_MERGE_BYTES = 1024 * MiB; | |
| 31 | + | |
| 32 | +/** The limits a plan keeps to (smaller ones let tests exercise every path with little data). */ | |
| 33 | +export type ComposeLimits = { stepOps: number; maxMergeBytes: number }; | |
| 34 | +export const COMPOSE_LIMITS: ComposeLimits = { stepOps: STEP_OPS, maxMergeBytes: MAX_MERGE_BYTES }; | |
| 35 | + | |
| 36 | +export function finalKey(b: BlobRef): string { | |
| 37 | + return runtimeObjectKey(b); | |
| 38 | +} | |
| 39 | + | |
| 40 | +export function stagingPrefix(b: BlobRef): string { | |
| 41 | + return `${STAGING_ROOT}${b.repo}/${b.kind}-${b.id}/`; | |
| 42 | +} | |
| 43 | + | |
| 44 | +/** Staged object for an Azure block id (any string the client chose). */ | |
| 45 | +export function blockKey(b: BlobRef, blockId: string): string { | |
| 46 | + return `${stagingPrefix(b)}b-${b64url(new TextEncoder().encode(blockId))}`; | |
| 47 | +} | |
| 48 | + | |
| 49 | +/** Staged object for a legacy cache chunk starting at `offset`. */ | |
| 50 | +export function chunkKey(b: BlobRef, offset: number): string { | |
| 51 | + return `${stagingPrefix(b)}o-${String(offset).padStart(16, "0")}`; | |
| 52 | +} | |
| 53 | + | |
| 54 | +/** Block ids of a Put Block List body, in order. */ | |
| 55 | +export function parseBlockList(xml: string): string[] { | |
| 56 | + const ids: string[] = []; | |
| 57 | + const re = /<(Latest|Committed|Uncommitted)>([^<]*)<\/\1>/g; | |
| 58 | + for (let m = re.exec(xml); m; m = re.exec(xml)) ids.push(decodeXml(m[2]!.trim())); | |
| 59 | + return ids; | |
| 60 | +} | |
| 61 | + | |
| 62 | +function decodeXml(s: string): string { | |
| 63 | + return s.replace(/&(lt|gt|amp|quot|apos);/g, (_, e: string) => ({ lt: "<", gt: ">", amp: "&", quot: '"', apos: "'" })[e]!); | |
| 64 | +} | |
| 65 | + | |
| 66 | +/** `bytes=a-b`, `bytes=a-` or `bytes=-n`, clamped to `size`. */ | |
| 67 | +export function parseRange(header: string | null | undefined, size: number): { offset: number; length: number } | null | "invalid" { | |
| 68 | + if (!header) return null; | |
| 69 | + const m = /^bytes=(\d*)-(\d*)$/.exec(header.trim()); | |
| 70 | + if (!m || (m[1] === "" && m[2] === "")) return "invalid"; | |
| 71 | + let start: number; | |
| 72 | + let end: number; | |
| 73 | + if (m[1] === "") { | |
| 74 | + const n = Number(m[2]); | |
| 75 | + start = Math.max(0, size - n); | |
| 76 | + end = size - 1; | |
| 77 | + } else { | |
| 78 | + start = Number(m[1]); | |
| 79 | + end = m[2] === "" ? size - 1 : Math.min(Number(m[2]), size - 1); | |
| 80 | + } | |
| 81 | + if (start >= size || end < start) return "invalid"; | |
| 82 | + return { offset: start, length: end - start + 1 }; | |
| 83 | +} | |
| 84 | + | |
| 85 | +type Body = ReadableStream | ArrayBuffer | ArrayBufferView; | |
| 86 | +type Meta = { httpMetadata?: { contentType?: string } }; | |
| 87 | +export interface MultipartHandle { | |
| 88 | + uploadId: string; | |
| 89 | + uploadPart(partNumber: number, value: Body): Promise<{ partNumber: number; etag: string }>; | |
| 90 | + complete(parts: Array<{ partNumber: number; etag: string }>): Promise<unknown>; | |
| 91 | + abort(): Promise<void>; | |
| 92 | +} | |
| 93 | + | |
| 94 | +/** The subset of R2Bucket used here (lets tests use an in-memory bucket). */ | |
| 95 | +export interface BlobBucket { | |
| 96 | + get(key: string, options?: { range?: { offset: number; length: number } }): Promise<{ body: ReadableStream; size: number } | null>; | |
| 97 | + head(key: string): Promise<{ size: number; etag: string } | null>; | |
| 98 | + put(key: string, value: Body, options?: Meta): Promise<{ size: number; etag: string } | null>; | |
| 99 | + list(options: { prefix: string; cursor?: string; limit?: number }): Promise<{ objects: Array<{ key: string; size: number }>; truncated: boolean; cursor?: string }>; | |
| 100 | + delete(keys: string | string[]): Promise<void>; | |
| 101 | + createMultipartUpload(key: string, options?: Meta): Promise<MultipartHandle>; | |
| 102 | + resumeMultipartUpload(key: string, uploadId: string): MultipartHandle; | |
| 103 | +} | |
| 104 | + | |
| 105 | +export async function listStaged(bucket: BlobBucket, b: BlobRef): Promise<Map<string, number>> { | |
| 106 | + const out = new Map<string, number>(); | |
| 107 | + let cursor: string | undefined; | |
| 108 | + do { | |
| 109 | + const page = await bucket.list({ prefix: stagingPrefix(b), cursor }); | |
| 110 | + for (const o of page.objects) out.set(o.key, o.size); | |
| 111 | + cursor = page.truncated ? page.cursor : undefined; | |
| 112 | + } while (cursor); | |
| 113 | + return out; | |
| 114 | +} | |
| 115 | + | |
| 116 | +export async function deleteKeys(bucket: BlobBucket, keys: string[]): Promise<void> { | |
| 117 | + for (let i = 0; i < keys.length; i += 1000) await bucket.delete(keys.slice(i, i + 1000)); | |
| 118 | +} | |
| 119 | + | |
| 120 | +/** Removes what an upload staged (the first `pages` listing pages; the lifecycle rule gets the rest). */ | |
| 121 | +export async function dropStaged(bucket: BlobBucket, b: BlobRef, pages = 1): Promise<void> { | |
| 122 | + let cursor: string | undefined; | |
| 123 | + for (let i = 0; i < pages; i++) { | |
| 124 | + const page = await bucket.list({ prefix: stagingPrefix(b), cursor, limit: 1000 }); | |
| 125 | + if (page.objects.length) await bucket.delete(page.objects.map((o) => o.key)); | |
| 126 | + if (!page.truncated) return; | |
| 127 | + cursor = page.cursor; | |
| 128 | + } | |
| 129 | +} | |
| 130 | + | |
| 131 | +/** A stream R2 accepts as a body: one with a known length (FixedLengthStream also enforces it). */ | |
| 132 | +export function sized(body: ReadableStream, size: number): ReadableStream { | |
| 133 | + const F = (globalThis as { FixedLengthStream?: new (n: number) => TransformStream }).FixedLengthStream; | |
| 134 | + if (!F) return body; | |
| 135 | + const t = new F(size); | |
| 136 | + void body.pipeTo(t.writable).catch(() => {}); | |
| 137 | + return t.readable; | |
| 138 | +} | |
| 139 | + | |
| 140 | +export class BodyTooLarge extends Error {} | |
| 141 | + | |
| 142 | +/** | |
| 143 | + * Stores a request body of unknown length without buffering all of it: up to | |
| 144 | + * one part in memory, then an R2 multipart upload. Fails with BodyTooLarge (and | |
| 145 | + * stores nothing) once more than `max` bytes arrive. | |
| 146 | + */ | |
| 147 | +export async function putStream(bucket: BlobBucket, key: string, body: ReadableStream, max: number, options?: Meta, partSize = 8 * MiB): Promise<number> { | |
| 148 | + const reader = body.getReader(); | |
| 149 | + let buf = new Uint8Array(Math.min(partSize, Math.max(max, 0) + 1)); | |
| 150 | + let fill = 0; | |
| 151 | + let total = 0; | |
| 152 | + let upload: MultipartHandle | null = null; | |
| 153 | + const parts: Array<{ partNumber: number; etag: string }> = []; | |
| 154 | + try { | |
| 155 | + for (;;) { | |
| 156 | + const { done, value } = await reader.read(); | |
| 157 | + if (done) break; | |
| 158 | + let chunk = value as Uint8Array; | |
| 159 | + total += chunk.length; | |
| 160 | + if (total > max) throw new BodyTooLarge(`the body is larger than ${max} bytes`); | |
| 161 | + while (chunk.length) { | |
| 162 | + if (fill === buf.length) { | |
| 163 | + upload ??= await bucket.createMultipartUpload(key, options); | |
| 164 | + parts.push(await upload.uploadPart(parts.length + 1, buf)); | |
| 165 | + buf = new Uint8Array(partSize); | |
| 166 | + fill = 0; | |
| 167 | + } | |
| 168 | + const n = Math.min(chunk.length, buf.length - fill); | |
| 169 | + buf.set(chunk.subarray(0, n), fill); | |
| 170 | + fill += n; | |
| 171 | + chunk = chunk.subarray(n); | |
| 172 | + } | |
| 173 | + } | |
| 174 | + if (!upload) { | |
| 175 | + await bucket.put(key, buf.slice(0, fill), options); | |
| 176 | + return total; | |
| 177 | + } | |
| 178 | + if (fill) parts.push(await upload.uploadPart(parts.length + 1, buf.slice(0, fill))); | |
| 179 | + await upload.complete(parts); | |
| 180 | + return total; | |
| 181 | + } catch (e) { | |
| 182 | + await reader.cancel().catch(() => {}); | |
| 183 | + if (upload) await upload.abort().catch(() => {}); | |
| 184 | + throw e; | |
| 185 | + } | |
| 186 | +} | |
| 187 | + | |
| 188 | +// ---------------------------------------------------------------- composing staged blocks | |
| 189 | + | |
| 190 | +/** A byte range of a staged object. */ | |
| 191 | +export type Slice = { key: string; offset: number; length: number; whole: boolean }; | |
| 192 | + | |
| 193 | +/** | |
| 194 | + * One unit of work, small enough for one invocation: | |
| 195 | + * - `merge`: concatenate slices into one object (an intermediate, or the final object); | |
| 196 | + * - `parts`: upload multipart parts of the final object, each the concatenation of its slices. | |
| 197 | + */ | |
| 198 | +export type ComposeStep = { kind: "merge"; target: string; slices: Slice[]; length: number } | { kind: "parts"; key: string; uploadId: string; parts: Array<{ partNumber: number; slices: Slice[]; length: number }> }; | |
| 199 | + | |
| 200 | +type Seg = { key: string; size: number }; | |
| 201 | + | |
| 202 | +/** Cuts `segs` (concatenated) into consecutive ranges of `size` bytes (the last may be shorter). */ | |
| 203 | +function cut(segs: Seg[], size: number): Slice[][] { | |
| 204 | + const out: Slice[][] = []; | |
| 205 | + let cur: Slice[] = []; | |
| 206 | + let room = size; | |
| 207 | + for (const s of segs) { | |
| 208 | + let off = 0; | |
| 209 | + while (off < s.size) { | |
| 210 | + const n = Math.min(room, s.size - off); | |
| 211 | + cur.push({ key: s.key, offset: off, length: n, whole: off === 0 && n === s.size }); | |
| 212 | + off += n; | |
| 213 | + room -= n; | |
| 214 | + if (room === 0) { | |
| 215 | + out.push(cur); | |
| 216 | + cur = []; | |
| 217 | + room = size; | |
| 218 | + } | |
| 219 | + } | |
| 220 | + } | |
| 221 | + if (cur.length) out.push(cur); | |
| 222 | + return out; | |
| 223 | +} | |
| 224 | + | |
| 225 | +const sliceBytes = (sl: Slice[]) => sl.reduce((n, s) => n + s.length, 0); | |
| 226 | + | |
| 227 | +/** | |
| 228 | + * Plans writing the concatenation of `segments` to `key` as steps that each make | |
| 229 | + * at most {@link STEP_OPS} R2 calls. Many small blocks (Go SDK uploads, up to | |
| 230 | + * Azure's 50,000) are first merged into larger intermediates under `scratch`, | |
| 231 | + * in as many rounds as needed; then the final object is written in one step, or | |
| 232 | + * as multipart parts of equal size in batches. | |
| 233 | + */ | |
| 234 | +export function planCompose(segments: Seg[], scratch: string, limits: ComposeLimits = COMPOSE_LIMITS): { rounds: Array<Extract<ComposeStep, { kind: "merge" }>[]>; total: number; partSize: number; segs: Seg[]; parts: Slice[][]; single: boolean; intermediates: string[] } { | |
| 235 | + const total = segments.reduce((n, s) => n + s.size, 0); | |
| 236 | + const partSize = Math.max(8 * MiB, Math.ceil(total / (MAX_PARTS - 1) / MiB) * MiB); | |
| 237 | + const rounds: Array<Extract<ComposeStep, { kind: "merge" }>[]> = []; | |
| 238 | + const intermediates: string[] = []; | |
| 239 | + let segs = segments.filter((s) => s.size > 0); | |
| 240 | + const { stepOps, maxMergeBytes } = limits; | |
| 241 | + const fitsOneStep = () => segs.length <= stepOps - 1 && total <= maxMergeBytes; | |
| 242 | + const partsFit = () => cut(segs, partSize).every((p) => p.length <= stepOps - 1); | |
| 243 | + // Merged segments may reach a part's size, so that a part never needs more slices than a step allows. | |
| 244 | + const mergeCap = Math.max(maxMergeBytes, partSize); | |
| 245 | + while (!fitsOneStep() && !partsFit()) { | |
| 246 | + // Merge runs of consecutive small segments; segments already big enough stay as they are. | |
| 247 | + const round: Extract<ComposeStep, { kind: "merge" }>[] = []; | |
| 248 | + const next: Seg[] = []; | |
| 249 | + let group: Seg[] = []; | |
| 250 | + let bytes = 0; | |
| 251 | + const flush = () => { | |
| 252 | + if (group.length === 1) next.push(group[0]!); | |
| 253 | + else if (group.length > 1) { | |
| 254 | + const target = `${scratch}m${rounds.length}-${round.length}`; | |
| 255 | + round.push({ kind: "merge", target, slices: group.map((s) => ({ key: s.key, offset: 0, length: s.size, whole: true })), length: bytes }); | |
| 256 | + intermediates.push(target); | |
| 257 | + next.push({ key: target, size: bytes }); | |
| 258 | + } | |
| 259 | + group = []; | |
| 260 | + bytes = 0; | |
| 261 | + }; | |
| 262 | + for (const s of segs) { | |
| 263 | + if (group.length && (group.length >= stepOps - 1 || bytes + s.size > mergeCap)) flush(); | |
| 264 | + group.push(s); | |
| 265 | + bytes += s.size; | |
| 266 | + } | |
| 267 | + flush(); | |
| 268 | + if (!round.length) break; | |
| 269 | + rounds.push(round); | |
| 270 | + segs = next; | |
| 271 | + } | |
| 272 | + if (fitsOneStep()) return { rounds, total, partSize, segs, parts: [], single: true, intermediates }; | |
| 273 | + return { rounds, total, partSize, segs, parts: cut(segs, partSize), single: false, intermediates }; | |
| 274 | +} | |
| 275 | + | |
| 276 | +/** Groups consecutive parts into steps of at most {@link STEP_OPS} calls. */ | |
| 277 | +export function batchParts(parts: Slice[][], stepOps = STEP_OPS): Array<Array<{ partNumber: number; slices: Slice[]; length: number }>> { | |
| 278 | + const out: Array<Array<{ partNumber: number; slices: Slice[]; length: number }>> = []; | |
| 279 | + let cur: Array<{ partNumber: number; slices: Slice[]; length: number }> = []; | |
| 280 | + let ops = 0; | |
| 281 | + parts.forEach((slices, i) => { | |
| 282 | + const cost = slices.length + 1; | |
| 283 | + if (cur.length && ops + cost > stepOps) { | |
| 284 | + out.push(cur); | |
| 285 | + cur = []; | |
| 286 | + ops = 0; | |
| 287 | + } | |
| 288 | + cur.push({ partNumber: i + 1, slices, length: sliceBytes(slices) }); | |
| 289 | + ops += cost; | |
| 290 | + }); | |
| 291 | + if (cur.length) out.push(cur); | |
| 292 | + return out; | |
| 293 | +} | |
| 294 | + | |
| 295 | +/** The concatenation of `slices`, read one after another. */ | |
| 296 | +function concat(bucket: BlobBucket, slices: Slice[]): ReadableStream { | |
| 297 | + let i = 0; | |
| 298 | + let reader: ReadableStreamDefaultReader<Uint8Array> | null = null; | |
| 299 | + return new ReadableStream<Uint8Array>({ | |
| 300 | + async pull(ctl) { | |
| 301 | + for (;;) { | |
| 302 | + if (!reader) { | |
| 303 | + const s = slices[i++]; | |
| 304 | + if (!s) return ctl.close(); | |
| 305 | + const o = await bucket.get(s.key, s.whole ? undefined : { range: { offset: s.offset, length: s.length } }); | |
| 306 | + if (!o) throw new Error(`staged block ${s.key} is missing`); | |
| 307 | + reader = o.body.getReader() as ReadableStreamDefaultReader<Uint8Array>; | |
| 308 | + } | |
| 309 | + const { done, value } = await reader.read(); | |
| 310 | + if (done) { | |
| 311 | + reader = null; | |
| 312 | + continue; | |
| 313 | + } | |
| 314 | + ctl.enqueue(value); | |
| 315 | + return; | |
| 316 | + } | |
| 317 | + }, | |
| 318 | + async cancel() { | |
| 319 | + await reader?.cancel().catch(() => {}); | |
| 320 | + }, | |
| 321 | + }); | |
| 322 | +} | |
| 323 | + | |
| 324 | +/** Runs one step against the bucket. Returns the parts it uploaded. */ | |
| 325 | +export async function runComposeStep(bucket: BlobBucket, step: ComposeStep, contentType?: string): Promise<Array<{ partNumber: number; etag: string }>> { | |
| 326 | + if (step.kind === "merge") { | |
| 327 | + await bucket.put(step.target, sized(concat(bucket, step.slices), step.length), contentType ? { httpMetadata: { contentType } } : undefined); | |
| 328 | + return []; | |
| 329 | + } | |
| 330 | + const upload = bucket.resumeMultipartUpload(step.key, step.uploadId); | |
| 331 | + const out: Array<{ partNumber: number; etag: string }> = []; | |
| 332 | + for (const p of step.parts) out.push(await upload.uploadPart(p.partNumber, sized(concat(bucket, p.slices), p.length))); | |
| 333 | + return out; | |
| 334 | +} | |
| 335 | + | |
| 336 | +/** Every key a step reads or writes. */ | |
| 337 | +export function stepKeys(step: ComposeStep): string[] { | |
| 338 | + return step.kind === "merge" ? [step.target, ...step.slices.map((s) => s.key)] : [step.key, ...step.parts.flatMap((p) => p.slices.map((s) => s.key))]; | |
| 339 | +} | |
| 340 | + | |
| 341 | +/** | |
| 342 | + * Writes the concatenation of `segments` (staged objects, in order) to `key`, | |
| 343 | + * running each step through `run` (a Durable Object call in production, so every | |
| 344 | + * step gets an invocation, and its subrequests, of its own). Intermediates are | |
| 345 | + * written under `scratch` and removed afterwards. | |
| 346 | + */ | |
| 347 | +export async function compose( | |
| 348 | + bucket: BlobBucket, | |
| 349 | + key: string, | |
| 350 | + segments: Seg[], | |
| 351 | + contentType: string, | |
| 352 | + o: { scratch: string; run?: (step: ComposeStep) => Promise<Array<{ partNumber: number; etag: string }>>; concurrency?: number; limits?: ComposeLimits }, | |
| 353 | +): Promise<number> { | |
| 354 | + const run = o.run ?? ((step: ComposeStep) => runComposeStep(bucket, step, contentType)); | |
| 355 | + const httpMetadata = { contentType }; | |
| 356 | + const limits = o.limits ?? COMPOSE_LIMITS; | |
| 357 | + const plan = planCompose(segments, o.scratch, limits); | |
| 358 | + const pool = async <T>(items: T[], f: (t: T) => Promise<void>) => { | |
| 359 | + let next = 0; | |
| 360 | + await Promise.all(Array.from({ length: Math.min(o.concurrency ?? 4, items.length) }, async () => { | |
| 361 | + while (next < items.length) await f(items[next++]!); | |
| 362 | + })); | |
| 363 | + }; | |
| 364 | + try { | |
| 365 | + for (const round of plan.rounds) await pool(round, async (s) => void (await run(s))); | |
| 366 | + if (plan.total === 0) { | |
| 367 | + await bucket.put(key, new Uint8Array(0), { httpMetadata }); | |
| 368 | + return 0; | |
| 369 | + } | |
| 370 | + if (plan.single) { | |
| 371 | + await run({ kind: "merge", target: key, slices: plan.segs.map((s) => ({ key: s.key, offset: 0, length: s.size, whole: true })), length: plan.total }); | |
| 372 | + return plan.total; | |
| 373 | + } | |
| 374 | + const upload = await bucket.createMultipartUpload(key, { httpMetadata }); | |
| 375 | + try { | |
| 376 | + const etags: Array<{ partNumber: number; etag: string }> = []; | |
| 377 | + await pool(batchParts(plan.parts, limits.stepOps), async (parts) => void etags.push(...(await run({ kind: "parts", key, uploadId: upload.uploadId, parts })))); | |
| 378 | + etags.sort((a, b) => a.partNumber - b.partNumber); | |
| 379 | + await upload.complete(etags); | |
| 380 | + } catch (e) { | |
| 381 | + await upload.abort().catch(() => {}); | |
| 382 | + throw e; | |
| 383 | + } | |
| 384 | + return plan.total; | |
| 385 | + } finally { | |
| 386 | + if (plan.intermediates.length) await deleteKeys(bucket, plan.intermediates).catch(() => {}); | |
| 387 | + } | |
| 388 | +} | |
| 389 | + | |
| 390 | +/** Minimum multipart part size, exported for tests. */ | |
| 391 | +export const MIN_PART_SIZE = MIN_PART; |
addedapps/diffinery/src/lib/job-permissions.ts+180−0View file
| @@ -1,0 +1,180 @@ | ||
| 1 | +import type { Permissions, Workflow } from "@diffinery/workflow-yaml"; | |
| 2 | +import { atLeast, type Role } from "../repo/types.js"; | |
| 3 | + | |
| 4 | +/** | |
| 5 | + * GitHub's permission model for a job's GITHUB_TOKEN: the effective | |
| 6 | + * `permissions:` of each job, and which permission each API request needs. | |
| 7 | + * https://docs.github.com/actions/reference/workflow-syntax-for-github-actions#permissions | |
| 8 | + */ | |
| 9 | + | |
| 10 | +export const PERMISSION_SCOPES = [ | |
| 11 | + "actions", | |
| 12 | + "attestations", | |
| 13 | + "checks", | |
| 14 | + "contents", | |
| 15 | + "deployments", | |
| 16 | + "discussions", | |
| 17 | + "id-token", | |
| 18 | + "issues", | |
| 19 | + "models", | |
| 20 | + "packages", | |
| 21 | + "pages", | |
| 22 | + "pull-requests", | |
| 23 | + "repository-projects", | |
| 24 | + "security-events", | |
| 25 | + "statuses", | |
| 26 | +] as const; | |
| 27 | +export type PermissionScope = (typeof PERMISSION_SCOPES)[number]; | |
| 28 | +export type Access = "none" | "read" | "write"; | |
| 29 | +export type JobPermissions = Record<PermissionScope, Access>; | |
| 30 | + | |
| 31 | +const LEVEL: Record<Access, number> = { none: 0, read: 1, write: 2 }; | |
| 32 | +/** Scopes that only take some levels (id-token: write or none; models: read or none). */ | |
| 33 | +const CEILING: Partial<Record<PermissionScope, Access>> = { models: "read" }; | |
| 34 | + | |
| 35 | +const all = (f: (s: PermissionScope) => Access): JobPermissions => Object.fromEntries(PERMISSION_SCOPES.map((s) => [s, f(s)])) as JobPermissions; | |
| 36 | +const clamp = (s: PermissionScope, a: Access): Access => (CEILING[s] && LEVEL[a] > LEVEL[CEILING[s]!] ? CEILING[s]! : a); | |
| 37 | + | |
| 38 | +export const WRITE_ALL: JobPermissions = all((s) => clamp(s, "write")); | |
| 39 | +export const READ_ALL: JobPermissions = all((s) => (s === "id-token" ? "none" : "read")); | |
| 40 | +/** The repository default when a workflow sets no `permissions:`, set to read and write (GitHub's permissive default). */ | |
| 41 | +export const PERMISSIVE_DEFAULT: JobPermissions = all((s) => (s === "id-token" ? "none" : clamp(s, "write"))); | |
| 42 | +/** The repository default set to read-only (GitHub's restricted default): contents and packages, read. */ | |
| 43 | +export const RESTRICTED_DEFAULT: JobPermissions = all((s) => (s === "contents" || s === "packages" ? "read" : "none")); | |
| 44 | + | |
| 45 | +/** `permissions:` as written (`read-all`, `write-all`, `{}` or a mapping) to a full table; listed-out scopes get none. */ | |
| 46 | +export function resolvePermissions(p: Permissions): JobPermissions { | |
| 47 | + if (p === "write-all") return { ...WRITE_ALL }; | |
| 48 | + if (p === "read-all") return { ...READ_ALL }; | |
| 49 | + return all((s) => { | |
| 50 | + const v = p[s]; | |
| 51 | + if (v !== "read" && v !== "write") return "none"; | |
| 52 | + // id-token has no read level; `id-token: read` grants nothing. | |
| 53 | + if (s === "id-token" && v === "read") return "none"; | |
| 54 | + return clamp(s, v); | |
| 55 | + }); | |
| 56 | +} | |
| 57 | + | |
| 58 | +/** The lower of each scope, for capping a called workflow by its caller. */ | |
| 59 | +export function minPermissions(a: JobPermissions, b: JobPermissions): JobPermissions { | |
| 60 | + return all((s) => (LEVEL[a[s]] <= LEVEL[b[s]] ? a[s] : b[s])); | |
| 61 | +} | |
| 62 | + | |
| 63 | +/** The parts of an expanded run (repo/reusable.ts) the permission chain needs. */ | |
| 64 | +export interface PermissionTree { | |
| 65 | + workflow: Workflow; | |
| 66 | + calls: Map<string, { parent: string | null; job: { permissions?: Permissions }; workflow: Workflow }>; | |
| 67 | + parentOf: Map<string, string | null>; | |
| 68 | +} | |
| 69 | + | |
| 70 | +/** | |
| 71 | + * A job's effective token permissions. Job-level `permissions:` replace the | |
| 72 | + * workflow's; without either, the repository default applies (read and write, | |
| 73 | + * or read-only: the repository's Actions setting). | |
| 74 | + * In a reusable workflow, a job without `permissions:` inherits its caller's, | |
| 75 | + * and nothing it declares can exceed what the calling job has. Pull requests | |
| 76 | + * from another repository (a fork) never get write. | |
| 77 | + */ | |
| 78 | +export function permissionsForJob(tree: PermissionTree, key: string, o: { untrusted?: boolean; repoDefault?: "read" | "write" } = {}): JobPermissions { | |
| 79 | + const top = o.untrusted ? READ_ALL : WRITE_ALL; | |
| 80 | + const base = o.untrusted ? READ_ALL : o.repoDefault === "read" ? RESTRICTED_DEFAULT : PERMISSIVE_DEFAULT; | |
| 81 | + const wfOf = (call: string | null) => (call === null ? tree.workflow : tree.calls.get(call)!.workflow); | |
| 82 | + // What jobs inside `call` may have at most: the calling job's permissions. | |
| 83 | + const cap = (call: string | null): JobPermissions => (call === null ? top : callerJob(call)); | |
| 84 | + // A workflow's own level: its `permissions:`, or what it inherits. | |
| 85 | + const level = (call: string | null): JobPermissions => { | |
| 86 | + const declared = wfOf(call).permissions; | |
| 87 | + if (declared !== undefined) return minPermissions(resolvePermissions(declared), cap(call)); | |
| 88 | + return call === null ? base : callerJob(call); | |
| 89 | + }; | |
| 90 | + const callerJob = (call: string): JobPermissions => { | |
| 91 | + const site = tree.calls.get(call)!; | |
| 92 | + return site.job.permissions !== undefined ? minPermissions(resolvePermissions(site.job.permissions), cap(site.parent)) : level(site.parent); | |
| 93 | + }; | |
| 94 | + const parent = tree.parentOf.get(key) ?? null; | |
| 95 | + const declared = tree.workflow.jobs[key]?.permissions; | |
| 96 | + return declared !== undefined ? minPermissions(resolvePermissions(declared), cap(parent)) : level(parent); | |
| 97 | +} | |
| 98 | + | |
| 99 | +/** | |
| 100 | + * The repository-level role the token gets; requests are further limited per | |
| 101 | + * scope by {@link requestPermission}. A job token never administers or | |
| 102 | + * maintains the repository, so branch protection always applies to it. | |
| 103 | + */ | |
| 104 | +export function roleForPermissions(p: JobPermissions): Role { | |
| 105 | + return PERMISSION_SCOPES.some((s) => s !== "id-token" && p[s] === "write") ? "write" : "read"; | |
| 106 | +} | |
| 107 | + | |
| 108 | +export type Requirement = { scopes: PermissionScope[]; access: Exclude<Access, "none"> } | { metadata: true } | { forbidden: string }; | |
| 109 | + | |
| 110 | +const ADMIN_ONLY = new Set(["hooks", "collaborators", "keys", "invitations", "settings", "rulesets", "teams", "transfer", "autolinks", "vulnerability-alerts"]); | |
| 111 | +const BY_SEGMENT: Record<string, PermissionScope[]> = { | |
| 112 | + issues: ["issues", "pull-requests"], | |
| 113 | + labels: ["issues", "pull-requests"], | |
| 114 | + milestones: ["issues", "pull-requests"], | |
| 115 | + assignees: ["issues", "pull-requests"], | |
| 116 | + pulls: ["pull-requests"], | |
| 117 | + pull: ["pull-requests"], | |
| 118 | + statuses: ["statuses"], | |
| 119 | + "check-runs": ["checks"], | |
| 120 | + "check-suites": ["checks"], | |
| 121 | + deployments: ["deployments"], | |
| 122 | + environments: ["deployments"], | |
| 123 | + actions: ["actions"], | |
| 124 | + pages: ["pages"], | |
| 125 | + discussions: ["discussions"], | |
| 126 | + "code-scanning": ["security-events"], | |
| 127 | + "secret-scanning": ["security-events"], | |
| 128 | + dependabot: ["security-events"], | |
| 129 | + packages: ["packages"], | |
| 130 | + projects: ["repository-projects"], | |
| 131 | + attestations: ["attestations"], | |
| 132 | +}; | |
| 133 | +const CONTENTS = new Set(["contents", "commits", "commit", "compare", "git", "branches", "tags", "releases", "merges", "tarball", "zipball", "readme", "dispatches", "comments", "edit", "new", "merge-upstream", "languages", "contributors", "stats", "archive", "blob", "tree", "raw", "info", "git-upload-pack", "git-receive-pack"]); | |
| 134 | + | |
| 135 | +/** | |
| 136 | + * The permission a job token needs for a request. `min` is the role the route | |
| 137 | + * itself asks for: git fetches and LFS downloads are POSTs that only read. | |
| 138 | + */ | |
| 139 | +export function requestPermission(method: string, path: string, min: Role): Requirement { | |
| 140 | + const readOnlyPost = /\/git-upload-pack$|\/info\/lfs\/objects\/batch$/.test(path); | |
| 141 | + const safe = method === "GET" || method === "HEAD" || method === "OPTIONS"; | |
| 142 | + const access = atLeast(min, "triage") || (!safe && !readOnlyPost) ? "write" : "read"; | |
| 143 | + // uploads.github.com is served at /api/uploads (release assets). | |
| 144 | + const api = /^\/api\/(?:v3|uploads)\/repos\/[^/]+\/[^/]+(?:\/(.*))?$/.exec(path); | |
| 145 | + // Web and git paths are /<owner>/<repo>[.git]/... | |
| 146 | + const web = api ? null : /^\/[^/]+\/[^/]+(?:\/(.*))?$/.exec(path); | |
| 147 | + const rest = api ? (api[1] ?? "") : web ? (web[1] ?? "") : undefined; | |
| 148 | + if (rest === undefined) return { forbidden: "this endpoint is not available to a workflow's GITHUB_TOKEN" }; | |
| 149 | + const segs = rest.split("/").filter(Boolean); | |
| 150 | + const first = segs[0] ?? ""; | |
| 151 | + // The repository itself: metadata through the API; its code page on the web. | |
| 152 | + if (!first && !api && access === "read") return { scopes: ["contents"], access }; | |
| 153 | + if (!first) return access === "read" ? { metadata: true } : { forbidden: "the repository's settings cannot be changed with a workflow's GITHUB_TOKEN" }; | |
| 154 | + const adminOnly = | |
| 155 | + ADMIN_ONLY.has(first) || | |
| 156 | + (first === "actions" && /^(secrets|variables|runners|permissions|oidc)$/.test(segs[1] ?? "")) || | |
| 157 | + (first === "environments" && (access === "write" || /^(secrets|variables)$/.test(segs[2] ?? ""))) || | |
| 158 | + (first === "branches" && segs.slice(2).includes("protection")); | |
| 159 | + if (adminOnly) return { forbidden: "this endpoint is not available to a workflow's GITHUB_TOKEN" }; | |
| 160 | + if (first === "issues" && segs.length === 1) return { scopes: ["issues"], access }; | |
| 161 | + if (first === "pulls" && /^(merge|update-branch)$/.test(segs[2] ?? "")) return { scopes: ["contents"], access }; | |
| 162 | + if (first === "commits" && /^(statuses|status)$/.test(segs[2] ?? "")) return { scopes: ["statuses"], access }; | |
| 163 | + if (first === "commits" && /^(check-runs|check-suites)$/.test(segs[2] ?? "")) return { scopes: ["checks"], access }; | |
| 164 | + if (BY_SEGMENT[first]) return { scopes: BY_SEGMENT[first]!, access }; | |
| 165 | + if (CONTENTS.has(first) || first.endsWith(".git")) return { scopes: ["contents"], access }; | |
| 166 | + return access === "read" ? { metadata: true } : { forbidden: "this endpoint is not available to a workflow's GITHUB_TOKEN" }; | |
| 167 | +} | |
| 168 | + | |
| 169 | +/** A request that only reads: anyone may make it on a public repository, whatever the token's permissions. */ | |
| 170 | +export function readOnly(req: Requirement): boolean { | |
| 171 | + return "metadata" in req || ("scopes" in req && req.access === "read"); | |
| 172 | +} | |
| 173 | + | |
| 174 | +/** Throws-free check: null when allowed, else the reason for a 403. */ | |
| 175 | +export function denied(p: JobPermissions, req: Requirement): string | null { | |
| 176 | + if ("metadata" in req) return null; | |
| 177 | + if ("forbidden" in req) return req.forbidden; | |
| 178 | + if (req.scopes.some((s) => LEVEL[p[s]] >= LEVEL[req.access])) return null; | |
| 179 | + return `Resource not accessible by integration (the workflow's GITHUB_TOKEN needs permissions: ${req.scopes.join(" or ")}: ${req.access})`; | |
| 180 | +} |
addedapps/diffinery/src/lib/runtime-auth.ts+128−0View file
| @@ -1,0 +1,128 @@ | ||
| 1 | +import { b64url, fromB64, fromB64url, safeEqual } from "./util.js"; | |
| 2 | + | |
| 3 | +/** | |
| 4 | + * Credentials for GitHub's Actions runtime services (cache, artifacts) as the | |
| 5 | + * official toolkit expects them: | |
| 6 | + * | |
| 7 | + * - `ACTIONS_RUNTIME_TOKEN` is a JWT. `@actions/artifact` reads the run and job | |
| 8 | + * backend ids from its `scp` claim, and buildkit's gha cache reads `ac`, `exp` | |
| 9 | + * and `nbf`. We sign it with HMAC-SHA256 under a key derived from MASTER_KEY, | |
| 10 | + * scoped to one job of one repository, and it expires with the job. | |
| 11 | + * - Blob URLs (where the toolkit PUTs and GETs archives with the Azure SDK or | |
| 12 | + * plain HTTP) are signed the same way, Azure SAS style: `se` (expiry) and | |
| 13 | + * `sp` (r or w) plus `sig`. | |
| 14 | + */ | |
| 15 | + | |
| 16 | +export interface RuntimeClaims { | |
| 17 | + /** Repository id (routes the request to its Durable Object). */ | |
| 18 | + repo: string; | |
| 19 | + run: string; | |
| 20 | + job: string; | |
| 21 | + /** Unix seconds. */ | |
| 22 | + exp: number; | |
| 23 | +} | |
| 24 | + | |
| 25 | +const keys = new Map<string, Promise<CryptoKey>>(); | |
| 26 | + | |
| 27 | +function runtimeKey(masterKey: string | undefined): Promise<CryptoKey> { | |
| 28 | + if (!masterKey) throw new Error("MASTER_KEY is not configured"); | |
| 29 | + let k = keys.get(masterKey); | |
| 30 | + if (!k) { | |
| 31 | + k = (async () => { | |
| 32 | + const raw = await crypto.subtle.importKey("raw", fromB64(masterKey) as BufferSource, "HKDF", false, ["deriveKey"]); | |
| 33 | + return crypto.subtle.deriveKey( | |
| 34 | + { name: "HKDF", hash: "SHA-256", salt: new TextEncoder().encode("diffinery-actions-runtime-v1"), info: new TextEncoder().encode("hmac") }, | |
| 35 | + raw, | |
| 36 | + { name: "HMAC", hash: "SHA-256", length: 256 }, | |
| 37 | + false, | |
| 38 | + ["sign"], | |
| 39 | + ); | |
| 40 | + })(); | |
| 41 | + keys.set(masterKey, k); | |
| 42 | + } | |
| 43 | + return k; | |
| 44 | +} | |
| 45 | + | |
| 46 | +async function hmac(masterKey: string | undefined, data: string): Promise<string> { | |
| 47 | + const sig = await crypto.subtle.sign("HMAC", await runtimeKey(masterKey), new TextEncoder().encode(data)); | |
| 48 | + return b64url(new Uint8Array(sig)); | |
| 49 | +} | |
| 50 | + | |
| 51 | +const enc = (o: unknown) => b64url(new TextEncoder().encode(JSON.stringify(o))); | |
| 52 | + | |
| 53 | +/** | |
| 54 | + * Mints the job's ACTIONS_RUNTIME_TOKEN. `cacheScopes` are the refs whose caches | |
| 55 | + * the job may read (the first one is also writable), in GitHub's `ac` format. | |
| 56 | + */ | |
| 57 | +export async function signRuntimeToken(masterKey: string | undefined, issuer: string, c: RuntimeClaims, cacheScopes: string[]): Promise<string> { | |
| 58 | + const nowS = Math.floor(Date.now() / 1000); | |
| 59 | + const ac = cacheScopes.map((s, i) => ({ Scope: s, Permission: i === 0 ? 3 : 1 })); | |
| 60 | + const payload = { | |
| 61 | + iss: issuer, | |
| 62 | + aud: "diffinery-actions-runtime", | |
| 63 | + nbf: nowS - 60, | |
| 64 | + iat: nowS, | |
| 65 | + exp: c.exp, | |
| 66 | + scp: `Actions.GenericRead:${c.run} Actions.Results:${c.run}:${c.job} Actions.UploadArtifacts:${c.run}:${c.job}`, | |
| 67 | + ac: JSON.stringify(ac), | |
| 68 | + repo: c.repo, | |
| 69 | + run: c.run, | |
| 70 | + job: c.job, | |
| 71 | + }; | |
| 72 | + const head = enc({ alg: "HS256", typ: "JWT" }); | |
| 73 | + const body = enc(payload); | |
| 74 | + return `${head}.${body}.${await hmac(masterKey, `${head}.${body}`)}`; | |
| 75 | +} | |
| 76 | + | |
| 77 | +/** Verifies a runtime token; returns its claims, or null when it is invalid or expired. */ | |
| 78 | +export async function verifyRuntimeToken(masterKey: string | undefined, token: string): Promise<RuntimeClaims | null> { | |
| 79 | + const [h, p, s] = token.split("."); | |
| 80 | + if (!h || !p || !s) return null; | |
| 81 | + let expected: string; | |
| 82 | + try { | |
| 83 | + expected = await hmac(masterKey, `${h}.${p}`); | |
| 84 | + } catch { | |
| 85 | + return null; | |
| 86 | + } | |
| 87 | + if (!safeEqual(expected, s)) return null; | |
| 88 | + try { | |
| 89 | + const head = JSON.parse(new TextDecoder().decode(fromB64url(h))) as { alg?: string }; | |
| 90 | + if (head.alg !== "HS256") return null; | |
| 91 | + const c = JSON.parse(new TextDecoder().decode(fromB64url(p))) as Partial<RuntimeClaims>; | |
| 92 | + if (typeof c.repo !== "string" || typeof c.run !== "string" || typeof c.job !== "string" || typeof c.exp !== "number") return null; | |
| 93 | + if (c.exp < Math.floor(Date.now() / 1000)) return null; | |
| 94 | + return { repo: c.repo, run: c.run, job: c.job, exp: c.exp }; | |
| 95 | + } catch { | |
| 96 | + return null; | |
| 97 | + } | |
| 98 | +} | |
| 99 | + | |
| 100 | +export type BlobKind = "cache" | "artifact"; | |
| 101 | + | |
| 102 | +export interface BlobRef { | |
| 103 | + repo: string; | |
| 104 | + kind: BlobKind; | |
| 105 | + id: number; | |
| 106 | +} | |
| 107 | + | |
| 108 | +const blobPayload = (b: BlobRef, perm: "r" | "w", exp: number) => `blob|${b.repo}|${b.kind}|${b.id}|${perm}|${exp}`; | |
| 109 | + | |
| 110 | +/** A signed URL for reading (`r`) or writing (`w`) a cache entry's or artifact's content. */ | |
| 111 | +export async function signBlobUrl(masterKey: string | undefined, origin: string, b: BlobRef, perm: "r" | "w", exp: number, fileName?: string): Promise<string> { | |
| 112 | + const sig = await hmac(masterKey, blobPayload(b, perm, exp)); | |
| 113 | + // A trailing file name lets clients that look at the URL path (e.g. `.zip`) see the type. | |
| 114 | + const tail = fileName ? `/${encodeURIComponent(fileName)}` : ""; | |
| 115 | + return `${origin}/_runtime/blob/${b.repo}/${b.kind}/${b.id}${tail}?sv=2025-01-05&se=${exp}&sp=${perm}&sig=${sig}`; | |
| 116 | +} | |
| 117 | + | |
| 118 | +/** Checks a blob URL's signature and expiry for the wanted permission. */ | |
| 119 | +export async function verifyBlobUrl(masterKey: string | undefined, b: BlobRef, perm: "r" | "w", q: URLSearchParams): Promise<boolean> { | |
| 120 | + const exp = Number(q.get("se")); | |
| 121 | + const sig = q.get("sig") ?? ""; | |
| 122 | + if (q.get("sp") !== perm || !Number.isInteger(exp) || exp < Math.floor(Date.now() / 1000)) return false; | |
| 123 | + try { | |
| 124 | + return safeEqual(await hmac(masterKey, blobPayload(b, perm, exp)), sig); | |
| 125 | + } catch { | |
| 126 | + return false; | |
| 127 | + } | |
| 128 | +} |
addedapps/diffinery/src/lib/storage-keys.ts+104−0View file
| @@ -1,0 +1,104 @@ | ||
| 1 | +/** | |
| 2 | + * Where things live in the R2 bucket. Durable and transient data have their own | |
| 3 | + * top-level prefixes, because R2 lifecycle rules (scripts/bootstrap.sh) can only | |
| 4 | + * select objects by prefix: | |
| 5 | + * | |
| 6 | + * | prefix | what | lifecycle rule | | |
| 7 | + * |----------------------|---------------------------------------------|---------------------------| | |
| 8 | + * | `repos/` | git objects, LFS, release assets | none: kept until deleted | | |
| 9 | + * | `registry/` | container registry blobs and manifests | none (store.ts sweeps) | | |
| 10 | + * | `artifacts/` | workflow artifacts | deleted after 90 days | | |
| 11 | + * | `actions-cache/` | Actions caches | none: evicted by the code | | |
| 12 | + * | `logs/` | archived job logs | deleted after 90 days | | |
| 13 | + * | `actions-staging/` | uploads in progress (blocks, chunks) | deleted after 1 day | | |
| 14 | + * | `archives/` | zipball/tarball cache (rebuilt on a miss) | deleted after 30 days | | |
| 15 | + * | `events/` | the event archive | deleted after 400 days | | |
| 16 | + * | |
| 17 | + * Every prefix also aborts multipart uploads left unfinished for a day. | |
| 18 | + * | |
| 19 | + * Artifacts, caches and logs were first stored under `repos/<id>/...` (and an | |
| 20 | + * age rule on `repos/` would have deleted git objects with them); reads fall back | |
| 21 | + * to those keys, listed after the current one. | |
| 22 | + * | |
| 23 | + * Deleting a repository removes its objects under the transient prefixes | |
| 24 | + * ({@link transientPrefixes}); nothing else would remove its caches. | |
| 25 | + */ | |
| 26 | + | |
| 27 | +export const STAGING_ROOT = "actions-staging/"; | |
| 28 | + | |
| 29 | +/** The object of an Actions runtime cache entry or artifact, by its numeric id. */ | |
| 30 | +export function runtimeObjectKey(b: { repo: string; kind: "cache" | "artifact"; id: number }): string { | |
| 31 | + return b.kind === "cache" ? `actions-cache/${b.repo}/${b.id}` : `artifacts/${b.repo}/${b.id}`; | |
| 32 | +} | |
| 33 | + | |
| 34 | +/** A cached source archive of a commit (zipball or tarball). */ | |
| 35 | +export function archiveKey(repo: string, sha: string, format: string): string { | |
| 36 | + return `archives/${repo}/${sha}.${format}`; | |
| 37 | +} | |
| 38 | + | |
| 39 | +/** A repository's prefixes outside repos/: Actions data and caches. */ | |
| 40 | +export function transientPrefixes(repo: string): string[] { | |
| 41 | + return [`actions-cache/${repo}/`, `artifacts/${repo}/`, `logs/${repo}/`, `${STAGING_ROOT}${repo}/`, `archives/${repo}/`]; | |
| 42 | +} | |
| 43 | + | |
| 44 | +type ListDelete = { | |
| 45 | + list(o: { prefix: string; cursor?: string; limit?: number }): Promise<{ objects: Array<{ key: string }>; truncated: boolean; cursor?: string }>; | |
| 46 | + delete(keys: string | string[]): Promise<unknown>; | |
| 47 | +}; | |
| 48 | + | |
| 49 | +/** | |
| 50 | + * Deletes every object under `prefixes`, a page of 1000 at a time, making at most | |
| 51 | + * `maxCalls` bucket calls (a Durable Object invocation has 1000 subrequests). | |
| 52 | + * Returns whether everything went. | |
| 53 | + */ | |
| 54 | +export async function deletePrefixes(bucket: ListDelete, prefixes: string[], maxCalls = 900): Promise<boolean> { | |
| 55 | + let calls = 0; | |
| 56 | + for (const prefix of prefixes) { | |
| 57 | + for (;;) { | |
| 58 | + if (calls + 2 > maxCalls) return false; | |
| 59 | + // Deleted objects leave the listing, so each page starts from the beginning. | |
| 60 | + const page = await bucket.list({ prefix, limit: 1000 }); | |
| 61 | + calls++; | |
| 62 | + if (!page.objects.length) break; | |
| 63 | + await bucket.delete(page.objects.map((o) => o.key)); | |
| 64 | + calls++; | |
| 65 | + if (!page.truncated) break; | |
| 66 | + } | |
| 67 | + } | |
| 68 | + return true; | |
| 69 | +} | |
| 70 | + | |
| 71 | +/** An archived job log: the current key, then the old layout's. */ | |
| 72 | +export function logKeys(repo: string, jobId: string): [string, string] { | |
| 73 | + return [`logs/${repo}/${jobId}.json`, `repos/${repo}/logs/${jobId}.json`]; | |
| 74 | +} | |
| 75 | + | |
| 76 | +/** A cache saved by an older runner (the pre-toolkit cache API): current key, then old. */ | |
| 77 | +export function legacyCacheKeys(repo: string, key: string): [string, string] { | |
| 78 | + const k = encodeURIComponent(key); | |
| 79 | + return [`actions-cache/${repo}/legacy/${k}`, `repos/${repo}/cache/${k}`]; | |
| 80 | +} | |
| 81 | + | |
| 82 | +/** An artifact uploaded by an older runner (a tarball per name): current key, then old. */ | |
| 83 | +export function legacyArtifactKeys(repo: string, runId: string, name: string): [string, string] { | |
| 84 | + const n = encodeURIComponent(name); | |
| 85 | + return [`artifacts/${repo}/legacy/${runId}/${n}`, `repos/${repo}/artifacts/${runId}/${n}`]; | |
| 86 | +} | |
| 87 | + | |
| 88 | +/** The first of `keys` that exists. */ | |
| 89 | +export async function getFirst(bucket: R2Bucket, keys: readonly string[]): Promise<R2ObjectBody | null> { | |
| 90 | + for (const k of keys) { | |
| 91 | + const o = await bucket.get(k); | |
| 92 | + if (o) return o; | |
| 93 | + } | |
| 94 | + return null; | |
| 95 | +} | |
| 96 | + | |
| 97 | +/** The first of `keys` that exists, without its body. */ | |
| 98 | +export async function headFirst(bucket: R2Bucket, keys: readonly string[]): Promise<R2Object | null> { | |
| 99 | + for (const k of keys) { | |
| 100 | + const o = await bucket.head(k); | |
| 101 | + if (o) return o; | |
| 102 | + } | |
| 103 | + return null; | |
| 104 | +} |
modifiedapps/diffinery/src/lib/util.ts+1−1View file
| @@ -67,7 +67,7 @@ | ||
| 67 | 67 | } |
| 68 | 68 | |
| 69 | 69 | /** Top-level paths the app serves; no account may use them as its login. */ |
| 70 | -export const RESERVED_LOGINS = new Set(["api", "mcp", "runner", "settings", "new", "notifications", "invite", "request-access", "healthz", "login", "logout", "admin", "explore", "favicon.ico", "robots.txt", "about", "docs", "static", "assets", "www", "git", "diffinery", "auth", "signin", "signup", "security", "v2", "-"]); | |
| 70 | +export const RESERVED_LOGINS = new Set(["api", "mcp", "runner", "settings", "new", "notifications", "invite", "request-access", "healthz", "twirp", "login", "logout", "admin", "explore", "favicon.ico", "robots.txt", "about", "docs", "static", "assets", "www", "git", "diffinery", "auth", "signin", "signup", "security", "v2", "-"]); | |
| 71 | 71 | |
| 72 | 72 | export const isValidLogin = (s: string): boolean => isValidName(s) && !s.includes(".") && !RESERVED_LOGINS.has(s.toLowerCase()); |
| 73 | 73 |
addedapps/diffinery/src/lib/webhook-payload.ts+29−0View file
| @@ -1,0 +1,29 @@ | ||
| 1 | +import type { DiffineryEvent } from "@diffinery/shared"; | |
| 2 | +import { ghRepo, nodeId, numericId } from "../http/gh.js"; | |
| 3 | +import type { Repo } from "./directory.js"; | |
| 4 | + | |
| 5 | +/** | |
| 6 | + * The JSON body of a webhook delivery, shaped like GitHub's. `row` is the | |
| 7 | + * repository's directory row; it gives the full repository object with the real | |
| 8 | + * `private` and `visibility`. Without it (a deleted repository) the object is | |
| 9 | + * minimal and reported as private. | |
| 10 | + */ | |
| 11 | +export function webhookPayload(publicUrl: string, ev: Exclude<DiffineryEvent, { type: "audit" }>, row: Repo | null) { | |
| 12 | + const repository = row | |
| 13 | + ? ghRepo(publicUrl, row) | |
| 14 | + : { id: numericId(ev.repoId), node_id: nodeId("Repository", ev.repoId), name: ev.repo, full_name: `${ev.owner}/${ev.repo}`, owner: { login: ev.owner }, private: true, visibility: "private", html_url: `${publicUrl}/${ev.owner}/${ev.repo}`, clone_url: `${publicUrl}/${ev.owner}/${ev.repo}.git` }; | |
| 15 | + const sender = { id: ev.actor.id, login: ev.actor.login, type: ev.actor.kind === "agent" ? "Bot" : ev.actor.kind === "system" ? "Bot" : "User", on_behalf_of: ev.actor.onBehalfOf }; | |
| 16 | + const { type: _t, repoId: _r, owner: _o, repo: _n, actor: _a, ...rest } = ev; | |
| 17 | + const action = "action" in ev ? (ev.action === "merged" ? "closed" : ev.action) : undefined; | |
| 18 | + const extra: Record<string, unknown> = {}; | |
| 19 | + if (ev.type === "pull_request") { | |
| 20 | + extra.pull_request = { number: ev.number, html_url: `${publicUrl}/${ev.owner}/${ev.repo}/pull/${ev.number}`, head: { ref: ev.headRef, sha: ev.headSha }, base: { ref: ev.baseRef }, merged: ev.action === "merged" }; | |
| 21 | + } | |
| 22 | + if (ev.type === "issues" || ev.type === "issue_comment") extra.issue = { number: ev.number, html_url: `${publicUrl}/${ev.owner}/${ev.repo}/issues/${ev.number}` }; | |
| 23 | + // GitHub-shaped events (release, status, deployment, ...) carry their payload ready-made. | |
| 24 | + if ("payload" in ev) { | |
| 25 | + const { payload, at: _at, ...head } = rest as typeof rest & { payload: Record<string, unknown>; at: string }; | |
| 26 | + return { ...head, ...payload, action, repository, sender }; | |
| 27 | + } | |
| 28 | + return { ...rest, action, repository, sender, ...extra }; | |
| 29 | +} |
modifiedapps/diffinery/src/mail/api.ts+2−2View file
| @@ -66,7 +66,7 @@ | ||
| 66 | 66 | |
| 67 | 67 | mailApi.put("/repos/:owner/:repo/subscription", async (c) => { |
| 68 | 68 | const me = writer(c); |
| 69 | - const l = await loadRepo(c, c.req.param("owner"), c.req.param("repo"), "read"); | |
| 69 | + const l = await loadRepo(c, c.req.param("owner"), c.req.param("repo"), "read", { whileArchived: true }); | |
| 70 | 70 | const b = await json<{ subscribed?: boolean; ignored?: boolean }>(c); |
| 71 | 71 | const level: WatchLevel = b.ignored ? "ignore" : b.subscribed === false ? "participating" : "all"; |
| 72 | 72 | await new MailStore(c.env.DB).setWatch(me.id, l.repo.id, level); |
| @@ -75,7 +75,7 @@ | ||
| 75 | 75 | |
| 76 | 76 | mailApi.delete("/repos/:owner/:repo/subscription", async (c) => { |
| 77 | 77 | const me = writer(c); |
| 78 | - const l = await loadRepo(c, c.req.param("owner"), c.req.param("repo"), "read"); | |
| 78 | + const l = await loadRepo(c, c.req.param("owner"), c.req.param("repo"), "read", { whileArchived: true }); | |
| 79 | 79 | await new MailStore(c.env.DB).setWatch(me.id, l.repo.id, "participating"); |
| 80 | 80 | return c.body(null, 204); |
| 81 | 81 | }); |
modifiedapps/diffinery/src/mail/notify.ts+1−0View file
| @@ -287,6 +287,7 @@ | ||
| 287 | 287 | jobs = to([e.actorId], { kind: "agent_created", data: { agent: e.target } }); |
| 288 | 288 | break; |
| 289 | 289 | case "push.secret_bypassed": |
| 290 | + case "api.secret_bypassed": | |
| 290 | 291 | case "push.blocked_secret": { |
| 291 | 292 | const [owner, name] = (e.target ?? "").split(":")[0]!.split("/"); |
| 292 | 293 | if (!owner || !name) return; |
modifiedapps/diffinery/src/mail/web.tsx+2−2View file
| @@ -344,7 +344,7 @@ | ||
| 344 | 344 | |
| 345 | 345 | post(web, "/:owner/:repo/subscription", async (c, f) => { |
| 346 | 346 | const me = requireUser(c); |
| 347 | - const l = await loadRepo(c, prm(c, "owner"), prm(c, "repo"), "read"); | |
| 347 | + const l = await loadRepo(c, prm(c, "owner"), prm(c, "repo"), "read", { whileArchived: true }); | |
| 348 | 348 | const back = f.s("back") === "/settings/notifications" ? "/settings/notifications" : repoUrl(l.repo); |
| 349 | 349 | const level = WATCH_LEVELS.find((x) => x === f.s("level")) as WatchLevel | undefined; |
| 350 | 350 | if (!level) throw new HttpError(422, "pick participating, all or ignore"); |
| @@ -354,7 +354,7 @@ | ||
| 354 | 354 | |
| 355 | 355 | post(web, "/:owner/:repo/issues/:number/subscription", async (c, f) => { |
| 356 | 356 | const me = requireUser(c); |
| 357 | - const l = await loadRepo(c, prm(c, "owner"), prm(c, "repo"), "read"); | |
| 357 | + const l = await loadRepo(c, prm(c, "owner"), prm(c, "repo"), "read", { whileArchived: true }); | |
| 358 | 358 | const n = Number(prm(c, "number")); |
| 359 | 359 | if (!Number.isInteger(n) || n < 1) throw new HttpError(404, "not found"); |
| 360 | 360 | const on = f.s("subscribed") === "1"; |
modifiedapps/diffinery/src/registry/api.ts+1−1View file
| @@ -122,7 +122,7 @@ | ||
| 122 | 122 | if (b.repository === null || b.repository === "") patch.repoId = null; |
| 123 | 123 | else { |
| 124 | 124 | const name = b.repository.includes("/") ? b.repository.split("/").pop()! : b.repository; |
| 125 | - const { repo } = await loadRepo(c, owner.login, name, "admin"); | |
| 125 | + const { repo } = await loadRepo(c, owner.login, name, "admin", { whileArchived: true }); | |
| 126 | 126 | patch.repoId = repo.id; |
| 127 | 127 | } |
| 128 | 128 | } |
modifiedapps/diffinery/src/repo/RepoDO.ts+141−6View file
| @@ -39,6 +39,7 @@ | ||
| 39 | 39 | } from "../git/protocol.js"; |
| 40 | 40 | import { scanForSecrets, secretBypassReason } from "../git/secret-scan.js"; |
| 41 | 41 | import { HttpError, isValidRefName, now } from "../lib/util.js"; |
| 42 | +import { deletePrefixes, transientPrefixes } from "../lib/storage-keys.js"; | |
| 42 | 43 | import { Actions } from "./actions.js"; |
| 43 | 44 | import { Collab } from "./collab.js"; |
| 44 | 45 | import { Deployments } from "./deployments.js"; |
| @@ -48,6 +49,8 @@ | ||
| 48 | 49 | import { MergeQueue, QUEUE_BRANCH_PREFIX, normalizeMergeQueue } from "./queue.js"; |
| 49 | 50 | import { Releases } from "./releases.js"; |
| 50 | 51 | import { REPO_SCHEMA, migrateRepoSchema } from "./schema.js"; |
| 52 | +import { Runtime } from "./runtime.js"; | |
| 53 | +import { TokenPolicy } from "./token-policy.js"; | |
| 51 | 54 | import { searchIssues } from "./search.js"; |
| 52 | 55 | import { Social } from "./social.js"; |
| 53 | 56 | import { Statuses } from "./statuses.js"; |
| @@ -85,6 +88,8 @@ | ||
| 85 | 88 | readonly collab: Collab; |
| 86 | 89 | readonly actions: Actions; |
| 87 | 90 | readonly queue: MergeQueue; |
| 91 | + readonly runtime: Runtime; | |
| 92 | + readonly policy: TokenPolicy; | |
| 88 | 93 | private metaCache: RepoMeta | null = null; |
| 89 | 94 | |
| 90 | 95 | constructor(ctx: DurableObjectState, env: Env) { |
| @@ -97,6 +102,8 @@ | ||
| 97 | 102 | this.collab = new Collab(this); |
| 98 | 103 | this.actions = new Actions(this, env); |
| 99 | 104 | this.queue = new MergeQueue(this); |
| 105 | + this.runtime = new Runtime(this, env); | |
| 106 | + this.policy = new TokenPolicy(this); | |
| 100 | 107 | } |
| 101 | 108 | |
| 102 | 109 | // ---------------------------------------------------------------- meta |
| @@ -114,6 +121,8 @@ | ||
| 114 | 121 | } |
| 115 | 122 | |
| 116 | 123 | async init(meta: RepoMeta): Promise<void> { |
| 124 | + // A repository created now (not one whose meta is updated) starts with read-only workflow tokens. | |
| 125 | + if (!this.readMeta()) this.policy.initNewRepo(); | |
| 117 | 126 | this.sql.exec("INSERT OR REPLACE INTO meta (key, value) VALUES ('meta', ?)", JSON.stringify(meta)); |
| 118 | 127 | this.metaCache = meta; |
| 119 | 128 | this.store = new RepoObjectStore(this.sql, this.env.BUCKET, meta.repoId); |
| @@ -127,6 +136,9 @@ | ||
| 127 | 136 | |
| 128 | 137 | /** Wipe all storage (repository deletion). */ |
| 129 | 138 | async destroy(): Promise<void> { |
| 139 | + // Caches, artifacts, logs and archives live outside repos/<id>/; with the rows gone, nothing would remove them. | |
| 140 | + const repoId = this.readMeta()?.repoId; | |
| 141 | + if (repoId) await deletePrefixes(this.env.BUCKET, transientPrefixes(repoId)).catch((e) => console.error("could not delete the repository's Actions objects", e)); | |
| 130 | 142 | await this.ctx.storage.deleteAll(); |
| 131 | 143 | this.metaCache = null; |
| 132 | 144 | } |
| @@ -320,6 +332,9 @@ | ||
| 320 | 332 | if (ref.startsWith("refs/pull/")) return { ref, ok: false, reason: "refs/pull/ is managed by Diffinery for pull requests" }; |
| 321 | 333 | if (ref.startsWith(`refs/heads/${QUEUE_BRANCH_PREFIX}`)) return { ref, ok: false, reason: "merge queue branches are read-only" }; |
| 322 | 334 | if (!deleting && !this.store.has(newOid)) return { ref, ok: false, reason: "missing objects" }; |
| 335 | + // A workflow's GITHUB_TOKEN never adds or changes workflow files (GitHub's `workflows` permission). | |
| 336 | + const workflowEdit = await this.policy.refuseWorkflowEdit(caller, oldOid, newOid); | |
| 337 | + if (workflowEdit) return { ref, ok: false, reason: workflowEdit }; | |
| 323 | 338 | if (!ref.startsWith("refs/heads/")) { |
| 324 | 339 | if (ref.startsWith("refs/tags/") && oldOid !== ZERO_OID && !atLeast(caller.role, "maintain")) return { ref, ok: false, reason: "only maintainers may move tags" }; |
| 325 | 340 | return { ref, ok: true }; |
| @@ -365,6 +380,7 @@ | ||
| 365 | 380 | at, |
| 366 | 381 | }; |
| 367 | 382 | this.emit(ev); |
| 383 | + this.policy.recordPush(ref, newOid, caller); | |
| 368 | 384 | this.deploys.noteRefUpdate(ref, newOid, caller); |
| 369 | 385 | if (ref.startsWith("refs/heads/") && newOid !== ZERO_OID) { |
| 370 | 386 | const branch = ref.slice(11); |
| @@ -477,6 +493,27 @@ | ||
| 477 | 493 | } |
| 478 | 494 | |
| 479 | 495 | /** |
| 496 | + * Push protection for commits made without a Git client: the same scan as `git push`, so the | |
| 497 | + * web editor, the contents and commits APIs and MCP cannot be used to slip a credential in. | |
| 498 | + */ | |
| 499 | + private protectCommit(changes: readonly FileChange[], allowSecrets: string | undefined, caller: Caller): void { | |
| 500 | + const blobs = changes.filter((ch) => !ch.delete && ch.content && ch.content.length < 1024 * 1024).map((ch) => ({ oid: ch.path, content: ch.content! })); | |
| 501 | + const leaks = scanForSecrets(blobs); | |
| 502 | + if (!leaks.length) return; | |
| 503 | + const kinds = [...new Set(leaks.map((l) => l.kind))]; | |
| 504 | + const paths = [...new Set(leaks.map((l) => l.oid))].slice(0, 10); | |
| 505 | + const target = `${this.meta.owner}/${this.meta.name}`; | |
| 506 | + const reason = allowSecrets?.trim().slice(0, 200); | |
| 507 | + if (reason && caller.role === "admin" && caller.kind === "user") { | |
| 508 | + this.emit({ type: "audit", action: "api.secret_bypassed", actor: caller, target, detail: { kinds, paths, reason }, at: now() }); | |
| 509 | + return; | |
| 510 | + } | |
| 511 | + this.emit({ type: "audit", action: "api.blocked_secret", actor: caller, target, detail: { kinds, paths }, at: now() }); | |
| 512 | + const hint = reason ? " Only repository admins (people, not agents) can bypass push protection." : ""; | |
| 513 | + throw new HttpError(422, `push protection: possible secret detected (${kinds.join(", ")}) in ${paths.join(", ")}. Remove it, or rotate it and ask an admin.${hint}`); | |
| 514 | + } | |
| 515 | + | |
| 516 | + /** | |
| 480 | 517 | * Create a commit from file changes without a Git client (used by the web |
| 481 | 518 | * editor, the REST contents API and AI agents through MCP). |
| 482 | 519 | */ |
| @@ -488,6 +525,8 @@ | ||
| 488 | 525 | createBranchFrom?: string; |
| 489 | 526 | author?: { name: string; email: string }; |
| 490 | 527 | committer?: { name: string; email: string }; |
| 528 | + /** A human admin's reason for committing a detected secret anyway (like `git push -o allow-secrets=`). */ | |
| 529 | + allowSecrets?: string; | |
| 491 | 530 | }, caller: Caller): Promise<{ sha: string; branch: string }> { |
| 492 | 531 | const ref = `refs/heads/${input.branch}`; |
| 493 | 532 | let parent = this.getRef(ref); |
| @@ -500,6 +539,7 @@ | ||
| 500 | 539 | const why = ch.delete ? null : unsafeTreePath(ch.path); |
| 501 | 540 | if (why) throw new HttpError(422, `invalid path ${JSON.stringify(ch.path)}: ${why}`); |
| 502 | 541 | } |
| 542 | + this.protectCommit(input.changes, input.allowSecrets, caller); | |
| 503 | 543 | const baseTree = parent ? (await readCommit(this.store, parent)).tree : null; |
| 504 | 544 | const tree = await applyChanges(this.store, baseTree, input.changes); |
| 505 | 545 | const ts = Math.floor(Date.now() / 1000); |
| @@ -763,16 +803,111 @@ | ||
| 763 | 803 | plannedRunError(runId: string, message: string) { |
| 764 | 804 | return this.actions.plannedError(runId, message); |
| 765 | 805 | } |
| 766 | - cacheKeyFor(key: string) { | |
| 767 | - return this.actions.cacheR2Key(key); | |
| 806 | + cacheSaveKeys(...a: Parameters<Actions["cacheSaveKeys"]>) { | |
| 807 | + return this.actions.cacheSaveKeys(...a); | |
| 768 | 808 | } |
| 769 | - artifactKeyFor(jobId: string, name: string) { | |
| 770 | - return this.actions.artifactR2Key(this.actions.runIdForJob(jobId), name); | |
| 809 | + cacheReadKeys(...a: Parameters<Actions["cacheReadKeys"]>) { | |
| 810 | + return this.actions.cacheReadKeys(...a); | |
| 771 | 811 | } |
| 772 | - artifactKeyForRun(runId: string, name: string) { | |
| 773 | - return this.actions.artifactR2Key(runId, name); | |
| 812 | + artifactKeysFor(jobId: string, name: string) { | |
| 813 | + return this.actions.artifactR2Keys(this.actions.runIdForJob(jobId), name); | |
| 774 | 814 | } |
| 815 | + artifactQuota() { | |
| 816 | + return this.runtime.artifactQuota; | |
| 817 | + } | |
| 775 | 818 | |
| 819 | + // ---------------------------------------------------------------- RPC passthroughs (Actions runtime services) | |
| 820 | + // The cache and artifact services of repo/runtime.ts, called by http/actions-runtime.ts | |
| 821 | + // with the job named by a runtime token, and by the REST caches/artifacts routes. | |
| 822 | + | |
| 823 | + rtJob(...a: Parameters<Runtime["job"]>) { | |
| 824 | + return this.runtime.job(...a); | |
| 825 | + } | |
| 826 | + rtGrantFor(...a: Parameters<Runtime["grantFor"]>) { | |
| 827 | + return this.runtime.grantFor(...a); | |
| 828 | + } | |
| 829 | + rtCacheLookup(...a: Parameters<Runtime["cacheLookup"]>) { | |
| 830 | + return this.runtime.cacheLookup(...a); | |
| 831 | + } | |
| 832 | + rtCacheReserve(...a: Parameters<Runtime["cacheReserve"]>) { | |
| 833 | + return this.runtime.cacheReserve(...a); | |
| 834 | + } | |
| 835 | + rtCacheReservation(...a: Parameters<Runtime["cacheReservation"]>) { | |
| 836 | + return this.runtime.cacheReservation(...a); | |
| 837 | + } | |
| 838 | + rtCacheWritable(...a: Parameters<Runtime["cacheWritable"]>) { | |
| 839 | + return this.runtime.cacheWritable(...a); | |
| 840 | + } | |
| 841 | + rtCacheCommit(...a: Parameters<Runtime["cacheCommit"]>) { | |
| 842 | + return this.runtime.cacheCommit(...a); | |
| 843 | + } | |
| 844 | + rtCacheGet(...a: Parameters<Runtime["cacheGet"]>) { | |
| 845 | + return this.runtime.cacheGet(...a); | |
| 846 | + } | |
| 847 | + rtListCaches(...a: Parameters<Runtime["listCaches"]>) { | |
| 848 | + return this.runtime.listCaches(...a); | |
| 849 | + } | |
| 850 | + rtDeleteCaches(...a: Parameters<Runtime["deleteCaches"]>) { | |
| 851 | + return this.runtime.deleteCaches(...a); | |
| 852 | + } | |
| 853 | + rtCacheUsage(...a: Parameters<Runtime["cacheUsage"]>) { | |
| 854 | + return this.runtime.cacheUsage(...a); | |
| 855 | + } | |
| 856 | + rtArtifactCreate(...a: Parameters<Runtime["artifactCreate"]>) { | |
| 857 | + return this.runtime.artifactCreate(...a); | |
| 858 | + } | |
| 859 | + rtArtifactUpload(...a: Parameters<Runtime["artifactUpload"]>) { | |
| 860 | + return this.runtime.artifactUpload(...a); | |
| 861 | + } | |
| 862 | + rtArtifactWritable(...a: Parameters<Runtime["artifactWritable"]>) { | |
| 863 | + return this.runtime.artifactWritable(...a); | |
| 864 | + } | |
| 865 | + rtArtifactFinalize(...a: Parameters<Runtime["artifactFinalize"]>) { | |
| 866 | + return this.runtime.artifactFinalize(...a); | |
| 867 | + } | |
| 868 | + rtArtifactGet(...a: Parameters<Runtime["artifactGet"]>) { | |
| 869 | + return this.runtime.artifactGet(...a); | |
| 870 | + } | |
| 871 | + rtJobArtifacts(...a: Parameters<Runtime["jobArtifacts"]>) { | |
| 872 | + return this.runtime.jobArtifacts(...a); | |
| 873 | + } | |
| 874 | + rtRunArtifacts(...a: Parameters<Runtime["runArtifacts"]>) { | |
| 875 | + return this.runtime.runArtifacts(...a); | |
| 876 | + } | |
| 877 | + rtRepoArtifacts(...a: Parameters<Runtime["repoArtifacts"]>) { | |
| 878 | + return this.runtime.repoArtifacts(...a); | |
| 879 | + } | |
| 880 | + rtArtifactDelete(...a: Parameters<Runtime["artifactDelete"]>) { | |
| 881 | + return this.runtime.artifactDelete(...a); | |
| 882 | + } | |
| 883 | + rtArtifactDeleteByName(...a: Parameters<Runtime["artifactDeleteByName"]>) { | |
| 884 | + return this.runtime.artifactDeleteByName(...a); | |
| 885 | + } | |
| 886 | + workflowSettings() { | |
| 887 | + return this.policy.settings(); | |
| 888 | + } | |
| 889 | + setWorkflowSettings(...a: Parameters<TokenPolicy["setSettings"]>) { | |
| 890 | + return this.policy.setSettings(...a); | |
| 891 | + } | |
| 892 | + rtStage(...a: Parameters<Runtime["stage"]>) { | |
| 893 | + return this.runtime.stage(...a); | |
| 894 | + } | |
| 895 | + rtUnstage(...a: Parameters<Runtime["unstage"]>) { | |
| 896 | + return this.runtime.unstage(...a); | |
| 897 | + } | |
| 898 | + artifactRoom() { | |
| 899 | + return this.runtime.artifactRoom(); | |
| 900 | + } | |
| 901 | + rtCheckCommitSize(...a: Parameters<Runtime["checkCommitSize"]>) { | |
| 902 | + return this.runtime.checkCommitSize(...a); | |
| 903 | + } | |
| 904 | + rtCacheMissing(...a: Parameters<Runtime["cacheMissing"]>) { | |
| 905 | + return this.runtime.cacheMissing(...a); | |
| 906 | + } | |
| 907 | + rtComposeStep(...a: Parameters<Runtime["composeStep"]>) { | |
| 908 | + return this.runtime.composeStep(...a); | |
| 909 | + } | |
| 910 | + | |
| 776 | 911 | // ---------------------------------------------------------------- RPC passthroughs (GitHub-compatible extras) |
| 777 | 912 | // Releases, commit statuses, deployments and environments, issue sub-resources |
| 778 | 913 | // (labels, reactions, events, locking), the Git data API, search and misc. |
modifiedapps/diffinery/src/repo/actions.ts+297−104View file
| @@ -17,11 +17,18 @@ | ||
| 17 | 17 | } from "@diffinery/workflow-yaml"; |
| 18 | 18 | import { decryptSecret, encryptSecret } from "../lib/crypto.js"; |
| 19 | 19 | import { Directory } from "../lib/directory.js"; |
| 20 | +import { finalKey } from "../lib/blob-store.js"; | |
| 21 | +import { getFirst, headFirst, legacyArtifactKeys, legacyCacheKeys, logKeys } from "../lib/storage-keys.js"; | |
| 22 | +import { permissionsForJob, type JobPermissions } from "../lib/job-permissions.js"; | |
| 23 | +import { signRuntimeToken } from "../lib/runtime-auth.js"; | |
| 24 | +import { ghRepo, numericId } from "../http/gh.js"; | |
| 20 | 25 | import { HttpError, newId, now, randomToken, sha256 } from "../lib/util.js"; |
| 21 | 26 | import type { Env } from "../env.js"; |
| 22 | 27 | import type { RepoDO } from "./RepoDO.js"; |
| 28 | +import { completePayload } from "./event-payloads.js"; | |
| 23 | 29 | import { atLeast, type Caller } from "./types.js"; |
| 24 | 30 | import { ReusableError, expandReusable, localKey, parseCallRef, type Expanded, type Source } from "./reusable.js"; |
| 31 | +import { ancestors, combineResults, jobStatus, runnerLabels, type JobNode } from "./job-graph.js"; | |
| 25 | 32 | |
| 26 | 33 | const WORKFLOW_DIRS = [".github/workflows", ".diffinery/workflows"]; |
| 27 | 34 | const SUPPORTED_LABEL = /^(ubuntu-.*|linux|diffinery(-.*)?|self-hosted|x64|amd64)$/i; |
| @@ -47,6 +54,8 @@ | ||
| 47 | 54 | display_title: string; |
| 48 | 55 | /** Commit the run reports checks on, when it differs from `sha` (pull request runs). */ |
| 49 | 56 | head_sha: string | null; |
| 57 | + /** 1 when the code the run executes came from an agent; its jobs then act as an agent. */ | |
| 58 | + agent_origin?: number; | |
| 50 | 59 | }; |
| 51 | 60 | |
| 52 | 61 | export type JobRow = { |
| @@ -54,7 +63,8 @@ | ||
| 54 | 63 | run_id: string; |
| 55 | 64 | job_key: string; |
| 56 | 65 | name: string; |
| 57 | - status: "queued" | "waiting" | "in_progress" | "completed"; | |
| 66 | + /** pending: held back by its concurrency group until the group's running job finishes. */ | |
| 67 | + status: "queued" | "pending" | "waiting" | "in_progress" | "completed"; | |
| 58 | 68 | conclusion: Conclusion | "timed_out" | null; |
| 59 | 69 | matrix: string; |
| 60 | 70 | strategy: string; |
| @@ -69,6 +79,11 @@ | ||
| 69 | 79 | timeout_minutes: number; |
| 70 | 80 | /** runs-on labels for self-hosted jobs (JSON array); null for Diffinery-hosted containers. */ |
| 71 | 81 | labels: string | null; |
| 82 | + /** `continue-on-error` as evaluated for this matrix leg (0 or 1). */ | |
| 83 | + continue_on_error: number; | |
| 84 | + /** Job-level `concurrency` group as evaluated for this leg, and its cancel-in-progress (0 or 1). */ | |
| 85 | + concurrency_group: string | null; | |
| 86 | + concurrency_cancel: number; | |
| 72 | 87 | }; |
| 73 | 88 | |
| 74 | 89 | export type RunInfo = { |
| @@ -145,13 +160,6 @@ | ||
| 145 | 160 | summary: j.summary, |
| 146 | 161 | }); |
| 147 | 162 | |
| 148 | -const statusFor = (needs: NeedsResult[], cancelled: boolean): StatusFns => ({ | |
| 149 | - success: () => !cancelled && needs.every((n) => n.result === "success" || n.result === "skipped"), | |
| 150 | - failure: () => needs.some((n) => n.result === "failure"), | |
| 151 | - cancelled: () => cancelled, | |
| 152 | - always: () => true, | |
| 153 | -}); | |
| 154 | - | |
| 155 | 163 | /** |
| 156 | 164 | * GitHub Actions compatible run orchestration state. The Cloudflare Workflow |
| 157 | 165 | * (`RunWorkflow`) drives a run through these methods; the runner agent inside |
| @@ -193,9 +201,12 @@ | ||
| 193 | 201 | return (await this.loadWorkflows(sha)).map((w) => ({ path: w.path, name: w.workflow?.name ?? w.path.split("/").pop()!, errors: w.errors, triggers: w.workflow ? Object.keys(w.workflow.on) : [] })); |
| 194 | 202 | } |
| 195 | 203 | |
| 196 | - private repositoryPayload() { | |
| 204 | + /** The `repository` object of event payloads: GitHub's shape, with the repository's real visibility. */ | |
| 205 | + async repositoryPayload(): Promise<Record<string, unknown>> { | |
| 197 | 206 | const m = this.repo.meta; |
| 198 | - return { id: m.repoId, name: m.name, full_name: `${m.owner}/${m.name}`, owner: { login: m.owner }, default_branch: m.defaultBranch, private: true, html_url: `${this.env.PUBLIC_URL}/${m.owner}/${m.name}`, clone_url: `${this.env.PUBLIC_URL}/${m.owner}/${m.name}.git` }; | |
| 207 | + const row = await new Directory(this.env).repoById(m.repoId); | |
| 208 | + if (row) return ghRepo(this.env.PUBLIC_URL, row); | |
| 209 | + return { id: numericId(m.repoId), name: m.name, full_name: `${m.owner}/${m.name}`, owner: { login: m.owner }, default_branch: m.defaultBranch, private: true, visibility: "private", html_url: `${this.env.PUBLIC_URL}/${m.owner}/${m.name}`, clone_url: `${this.env.PUBLIC_URL}/${m.owner}/${m.name}.git` }; | |
| 199 | 210 | } |
| 200 | 211 | |
| 201 | 212 | // ---------------------------------------------------------------- triggers |
| @@ -216,7 +227,7 @@ | ||
| 216 | 227 | created: ev.created, |
| 217 | 228 | deleted: ev.deleted, |
| 218 | 229 | forced: ev.forced, |
| 219 | - repository: this.repositoryPayload(), | |
| 230 | + repository: await this.repositoryPayload(), | |
| 220 | 231 | pusher: { name: caller.login }, |
| 221 | 232 | sender: { login: caller.login, type: caller.kind === "agent" ? "Bot" : "User" }, |
| 222 | 233 | head_commit: { id: ev.after, message: head.message, timestamp: new Date(head.committer.timestamp * 1000).toISOString(), author: { name: head.author.name, email: head.author.email } }, |
| @@ -247,6 +258,7 @@ | ||
| 247 | 258 | const { baseSha, mergeSha } = pull.merged ? { baseSha: this.repo.getRef(`refs/heads/${pull.base}`), mergeSha: null } : await this.repo.collab.refreshPullRefs(number); |
| 248 | 259 | const changedFiles = pull.merged ? undefined : (await this.repo.collab.pullFiles(number)).files.map((f) => f.path); |
| 249 | 260 | const m = this.repo.meta; |
| 261 | + const repository = await this.repositoryPayload(); | |
| 250 | 262 | const payload = { |
| 251 | 263 | action, |
| 252 | 264 | number, |
| @@ -264,12 +276,12 @@ | ||
| 264 | 276 | user: { login: issue.author.login }, |
| 265 | 277 | labels: issue.labels.map((name) => ({ name })), |
| 266 | 278 | assignees: issue.assignees.map((login) => ({ login })), |
| 267 | - head: { ref: pull.head, sha: pull.headSha, repo: this.repositoryPayload() }, | |
| 268 | - base: { ref: pull.base, sha: baseSha, repo: this.repositoryPayload() }, | |
| 279 | + head: { ref: pull.head, sha: pull.headSha, repo: repository }, | |
| 280 | + base: { ref: pull.base, sha: baseSha, repo: repository }, | |
| 269 | 281 | html_url: `${this.env.PUBLIC_URL}/${m.owner}/${m.name}/pull/${number}`, |
| 270 | 282 | url: `${this.env.PUBLIC_URL}/api/v3/repos/${m.owner}/${m.name}/pulls/${number}`, |
| 271 | 283 | }, |
| 272 | - repository: this.repositoryPayload(), | |
| 284 | + repository, | |
| 273 | 285 | sender: { login: caller.login }, |
| 274 | 286 | }; |
| 275 | 287 | const common = { payload, caller, prNumber: number, headSha: pull.headSha, headRef: pull.head, baseRef: pull.base, title: issue.title }; |
| @@ -308,7 +320,7 @@ | ||
| 308 | 320 | committer: { name: head.committer.name, email: head.committer.email }, |
| 309 | 321 | }, |
| 310 | 322 | }, |
| 311 | - repository: this.repositoryPayload(), | |
| 323 | + repository: await this.repositoryPayload(), | |
| 312 | 324 | sender: { login: g.sender.login }, |
| 313 | 325 | }; |
| 314 | 326 | for (const w of await this.loadWorkflows(g.sha)) { |
| @@ -317,7 +329,7 @@ | ||
| 317 | 329 | continue; |
| 318 | 330 | } |
| 319 | 331 | if (!matchesTrigger(w.workflow, { name: "merge_group", action: "checks_requested", baseRef: g.base })) continue; |
| 320 | - await this.createRun(w.workflow, { event: "merge_group", ref: g.ref, sha: g.sha, payload, caller: g.sender, title: `Merge group for #${g.number}: ${g.title}` }); | |
| 332 | + await this.createRun(w.workflow, { event: "merge_group", ref: g.ref, sha: g.sha, payload, caller: g.sender, title: `Merge group for #${g.number}: ${g.title}`, prNumbers: [g.number] }); | |
| 321 | 333 | } |
| 322 | 334 | } |
| 323 | 335 | |
| @@ -343,7 +355,7 @@ | ||
| 343 | 355 | resolved[k] = d.type === "boolean" ? v === true || v === "true" : d.type === "number" ? Number(v) : v; |
| 344 | 356 | } |
| 345 | 357 | const fullRef = ref.startsWith("refs/") ? ref : this.repo.getRef(`refs/heads/${ref}`) ? `refs/heads/${ref}` : `refs/tags/${ref}`; |
| 346 | - const payload = { inputs: resolved, ref: fullRef, repository: this.repositoryPayload(), sender: { login: caller.login }, workflow: w.path }; | |
| 358 | + const payload = { inputs: resolved, ref: fullRef, repository: await this.repositoryPayload(), sender: { login: caller.login }, workflow: w.path }; | |
| 347 | 359 | const m = this.repo.meta; |
| 348 | 360 | this.repo.emit({ type: "workflow_dispatch", repoId: m.repoId, owner: m.owner, repo: m.name, ref: fullRef, workflow: w.path, inputs: resolved as Record<string, Json>, actor: { id: caller.id, login: caller.login, kind: caller.kind, onBehalfOf: caller.onBehalfOf }, at: now() }); |
| 349 | 361 | return (await this.createRun(w.workflow, { event: "workflow_dispatch", ref: fullRef, sha, payload, caller, inputs: resolved, title: w.workflow.name ?? w.path }))!; |
| @@ -355,7 +367,7 @@ | ||
| 355 | 367 | const w = (await this.loadWorkflows(sha)).find((x) => x.path === workflowPath); |
| 356 | 368 | if (!w?.workflow || !matchesTrigger(w.workflow, { name: "schedule", cron })) return; |
| 357 | 369 | const system: Caller = { id: "system", login: "diffinery-scheduler", kind: "system", role: "write" }; |
| 358 | - await this.createRun(w.workflow, { event: "schedule", ref: this.repo.headRef, sha, payload: { schedule: cron, repository: this.repositoryPayload() }, caller: system, title: w.workflow.name ?? w.path }); | |
| 370 | + await this.createRun(w.workflow, { event: "schedule", ref: this.repo.headRef, sha, payload: { schedule: cron, repository: await this.repositoryPayload() }, caller: system, title: w.workflow.name ?? w.path }); | |
| 359 | 371 | } |
| 360 | 372 | |
| 361 | 373 | private async syncSchedules(workflows: Array<{ path: string; workflow?: Workflow }>): Promise<void> { |
| @@ -394,7 +406,8 @@ | ||
| 394 | 406 | repositoryUrl: `${this.env.PUBLIC_URL}/${m.owner}/${m.name}.git`, |
| 395 | 407 | actor: actor.login, |
| 396 | 408 | triggering_actor: actor.login, |
| 397 | - run_id: run.id, | |
| 409 | + // Numeric like GitHub's (actions parse it, e.g. download-artifact's run-id); the API accepts it as a run id. | |
| 410 | + run_id: String(run.run_number), | |
| 398 | 411 | run_number: String(run.run_number), |
| 399 | 412 | run_attempt: String(run.attempt), |
| 400 | 413 | workflow: run.workflow_name, |
| @@ -440,8 +453,27 @@ | ||
| 440 | 453 | |
| 441 | 454 | async createRun( |
| 442 | 455 | wf: Workflow, |
| 443 | - o: { event: string; ref: string; sha: string; headSha?: string; payload: Record<string, unknown>; caller: Caller | { id: string; login: string; kind: Actor["kind"] }; prNumber?: number; inputs?: Record<string, unknown>; headRef?: string; baseRef?: string; title: string }, | |
| 456 | + o: { | |
| 457 | + event: string; | |
| 458 | + ref: string; | |
| 459 | + sha: string; | |
| 460 | + headSha?: string; | |
| 461 | + payload: Record<string, unknown>; | |
| 462 | + caller: Caller | { id: string; login: string; kind: Actor["kind"] }; | |
| 463 | + prNumber?: number; | |
| 464 | + inputs?: Record<string, unknown>; | |
| 465 | + headRef?: string; | |
| 466 | + baseRef?: string; | |
| 467 | + title: string; | |
| 468 | + /** A re-run of a run whose code came from an agent. */ | |
| 469 | + agentOrigin?: boolean; | |
| 470 | + /** Pull requests whose code the run includes (a merge group's). */ | |
| 471 | + prNumbers?: number[]; | |
| 472 | + }, | |
| 444 | 473 | ): Promise<RunInfo | null> { |
| 474 | + // Like GitHub, what a job does with its GITHUB_TOKEN (pushes, pull requests) starts no new | |
| 475 | + // runs, so workflows cannot trigger themselves in a loop; workflow_dispatch is the exception. | |
| 476 | + if (o.caller.id.startsWith("job:") && o.event !== "workflow_dispatch") return null; | |
| 445 | 477 | const id = newId(); |
| 446 | 478 | const t = now(); |
| 447 | 479 | const run: RunRow = { |
| @@ -459,11 +491,19 @@ | ||
| 459 | 491 | updated_at: t, |
| 460 | 492 | pr_number: o.prNumber ?? null, |
| 461 | 493 | attempt: 1, |
| 462 | - event_payload: JSON.stringify(o.payload), | |
| 494 | + event_payload: JSON.stringify(completePayload({ publicUrl: this.env.PUBLIC_URL, meta: this.repo.meta, visibility: (o.payload.repository as { visibility?: string } | undefined)?.visibility === "public" ? "public" : "private" }, o.payload, o.caller)), | |
| 463 | 495 | inputs: JSON.stringify(o.inputs ?? {}), |
| 464 | 496 | concurrency_group: null, |
| 465 | 497 | display_title: o.title, |
| 466 | 498 | head_sha: o.headSha && o.headSha !== o.sha ? o.headSha : null, |
| 499 | + agent_origin: this.repo.policy.agentOrigin({ | |
| 500 | + caller: o.caller, | |
| 501 | + ref: o.ref, | |
| 502 | + prNumbers: [...(o.prNumber !== undefined ? [o.prNumber] : []), ...(o.prNumbers ?? [])], | |
| 503 | + inherited: o.agentOrigin, | |
| 504 | + }) | |
| 505 | + ? 1 | |
| 506 | + : 0, | |
| 467 | 507 | }; |
| 468 | 508 | const ctx: ExprContext = { github: this.githubContext(run, "", ""), inputs: o.inputs ?? {}, vars: this.variables() }; |
| 469 | 509 | try { |
| @@ -473,7 +513,7 @@ | ||
| 473 | 513 | console.warn("run-name/concurrency evaluation failed", e); |
| 474 | 514 | } |
| 475 | 515 | this.sql.exec( |
| 476 | - "INSERT INTO runs (id, run_number, workflow_path, workflow_name, event, ref, sha, actor, status, created_at, updated_at, pr_number, attempt, event_payload, inputs, concurrency_group, display_title, head_sha) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)", | |
| 516 | + "INSERT INTO runs (id, run_number, workflow_path, workflow_name, event, ref, sha, actor, status, created_at, updated_at, pr_number, attempt, event_payload, inputs, concurrency_group, display_title, head_sha, agent_origin) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)", | |
| 477 | 517 | run.id, |
| 478 | 518 | run.run_number, |
| 479 | 519 | run.workflow_path, |
| @@ -492,6 +532,7 @@ | ||
| 492 | 532 | run.concurrency_group, |
| 493 | 533 | run.display_title, |
| 494 | 534 | run.head_sha, |
| 535 | + run.agent_origin ?? 0, | |
| 495 | 536 | ); |
| 496 | 537 | if (run.concurrency_group && wf.concurrency) { |
| 497 | 538 | const cancelInProgress = evaluateValue(wf.concurrency.cancelInProgress, ctx); |
| @@ -506,8 +547,11 @@ | ||
| 506 | 547 | return toRun(run); |
| 507 | 548 | } |
| 508 | 549 | |
| 550 | + /** A run by id, or by its number (github.run_id; the latest attempt). */ | |
| 509 | 551 | private runRow(id: string): RunRow { |
| 510 | - const r = this.sql.exec<RunRow>("SELECT * FROM runs WHERE id = ?", id).toArray()[0]; | |
| 552 | + const r = /^\d+$/.test(id) | |
| 553 | + ? this.sql.exec<RunRow>("SELECT * FROM runs WHERE run_number = ? ORDER BY attempt DESC, created_at DESC LIMIT 1", Number(id)).toArray()[0] | |
| 554 | + : this.sql.exec<RunRow>("SELECT * FROM runs WHERE id = ?", id).toArray()[0]; | |
| 511 | 555 | if (!r) throw new HttpError(404, "run not found"); |
| 512 | 556 | return r; |
| 513 | 557 | } |
| @@ -552,10 +596,12 @@ | ||
| 552 | 596 | } |
| 553 | 597 | |
| 554 | 598 | getRun(id: string): RunInfo & { jobs: JobInfo[] } { |
| 555 | - return { ...toRun(this.runRow(id)), jobs: this.jobs(id) }; | |
| 599 | + const run = this.runRow(id); | |
| 600 | + return { ...toRun(run), jobs: this.jobs(run.id) }; | |
| 556 | 601 | } |
| 557 | 602 | |
| 558 | 603 | jobs(runId: string): JobInfo[] { |
| 604 | + if (/^\d+$/.test(runId)) runId = this.runRow(runId).id; | |
| 559 | 605 | return this.sql.exec<JobRow>("SELECT * FROM jobs WHERE run_id = ? ORDER BY rowid", runId).toArray().map(toJob); |
| 560 | 606 | } |
| 561 | 607 | |
| @@ -567,34 +613,38 @@ | ||
| 567 | 613 | const j = this.jobRow(jobId); |
| 568 | 614 | const rows = this.sql.exec<LogLine & { seq: number }>("SELECT seq, t, step, text FROM logs WHERE job_id = ? AND seq > ? ORDER BY seq LIMIT 5000", jobId, afterSeq).toArray(); |
| 569 | 615 | if (rows.length === 0 && j.status === "completed") { |
| 570 | - const archived = await this.env.BUCKET.get(this.logKey(jobId)); | |
| 616 | + const archived = await getFirst(this.env.BUCKET, logKeys(this.repo.meta.repoId, jobId)); | |
| 571 | 617 | if (archived && afterSeq < 0) return { lines: (await archived.json()) as Array<LogLine & { seq: number }>, complete: true }; |
| 572 | 618 | } |
| 573 | 619 | return { lines: rows, complete: j.status === "completed" }; |
| 574 | 620 | } |
| 575 | 621 | |
| 576 | 622 | private logKey(jobId: string) { |
| 577 | - return `repos/${this.repo.meta.repoId}/logs/${jobId}.json`; | |
| 623 | + return logKeys(this.repo.meta.repoId, jobId)[0]; | |
| 578 | 624 | } |
| 579 | 625 | |
| 580 | 626 | // ---------------------------------------------------------------- orchestration (called by RunWorkflow) |
| 581 | 627 | |
| 582 | - async plan(runId: string): Promise<{ levels: string[][]; error?: string; concurrencyBlocked: boolean }> { | |
| 628 | + /** | |
| 629 | + * The run's job graph: each job with the jobs it waits for (reusable workflow calls | |
| 630 | + * expanded). RunWorkflow starts a job as soon as its own needs have finished. | |
| 631 | + */ | |
| 632 | + async plan(runId: string): Promise<{ jobs: JobNode[]; error?: string; concurrencyBlocked: boolean }> { | |
| 583 | 633 | const run = this.runRow(runId); |
| 584 | - if (run.status === "completed") return { levels: [], error: "run already completed", concurrencyBlocked: false }; | |
| 634 | + if (run.status === "completed") return { jobs: [], error: "run already completed", concurrencyBlocked: false }; | |
| 585 | 635 | let exp: Expanded; |
| 586 | 636 | try { |
| 587 | 637 | exp = await this.expanded(run); |
| 588 | 638 | } catch (e) { |
| 589 | - return { levels: [], error: (e as Error).message, concurrencyBlocked: false }; | |
| 639 | + return { jobs: [], error: (e as Error).message, concurrencyBlocked: false }; | |
| 590 | 640 | } |
| 591 | 641 | const { order, errors } = jobOrder(exp.workflow); |
| 592 | - if (errors.length) return { levels: [], error: errors.map((e) => e.message).join("; "), concurrencyBlocked: false }; | |
| 642 | + if (errors.length) return { jobs: [], error: errors.map((e) => e.message).join("; "), concurrencyBlocked: false }; | |
| 593 | 643 | const blocked = |
| 594 | 644 | !!run.concurrency_group && |
| 595 | 645 | this.sql.exec<{ n: number }>("SELECT count(*) AS n FROM runs WHERE concurrency_group = ? AND status = 'in_progress' AND id != ?", run.concurrency_group, runId).one().n > 0; |
| 596 | 646 | if (!blocked) this.sql.exec("UPDATE runs SET status = 'in_progress', updated_at = ? WHERE id = ? AND status = 'queued'", now(), runId); |
| 597 | - return { levels: order, concurrencyBlocked: blocked }; | |
| 647 | + return { jobs: order.flat().map((key) => ({ key, needs: exp.workflow.jobs[key]!.needs })), concurrencyBlocked: blocked }; | |
| 598 | 648 | } |
| 599 | 649 | |
| 600 | 650 | /** The run's workflow with reusable workflow calls expanded (see ./reusable.ts). */ |
| @@ -653,21 +703,27 @@ | ||
| 653 | 703 | return out; |
| 654 | 704 | } |
| 655 | 705 | |
| 706 | + /** Status functions for the `if:` of a job or call: from the results of all its ancestors, like GitHub. */ | |
| 707 | + private statusOf(run: RunRow, exp: Expanded, key: string): StatusFns { | |
| 708 | + const results = ancestors(exp.needs, exp.needs.get(key) ?? []).map((k) => combineResults(this.rowsOf(run, exp, k))); | |
| 709 | + return jobStatus(results, run.conclusion === "cancelled"); | |
| 710 | + } | |
| 711 | + | |
| 712 | + /** The job rows of a job, or of every job inside a reusable workflow call. */ | |
| 713 | + private rowsOf(run: RunRow, exp: Expanded, key: string): Array<JobRow & { continueOnError: boolean }> { | |
| 714 | + const rows = exp.calls.has(key) | |
| 715 | + ? this.sql.exec<JobRow>("SELECT * FROM jobs WHERE run_id = ? AND substr(job_key, 1, ?) = ?", run.id, key.length + 1, `${key}/`).toArray() | |
| 716 | + : this.sql.exec<JobRow>("SELECT * FROM jobs WHERE run_id = ? AND job_key = ?", run.id, key).toArray(); | |
| 717 | + return rows.map((r) => ({ ...r, continueOnError: !!r.continue_on_error })); | |
| 718 | + } | |
| 719 | + | |
| 656 | 720 | /** Result and outputs of a job, or of a reusable workflow call (from its `on.workflow_call.outputs`). */ |
| 657 | 721 | private async resultOf(run: RunRow, exp: Expanded, key: string): Promise<NeedsResult> { |
| 658 | 722 | const call = exp.calls.get(key); |
| 659 | - const rows = call | |
| 660 | - ? this.sql.exec<JobRow>("SELECT * FROM jobs WHERE run_id = ? AND substr(job_key, 1, ?) = ?", run.id, key.length + 1, `${key}/`).toArray() | |
| 661 | - : this.sql.exec<JobRow>("SELECT * FROM jobs WHERE run_id = ? AND job_key = ?", run.id, key).toArray(); | |
| 662 | - let result: Conclusion = rows.length === 0 ? "skipped" : "success"; | |
| 723 | + const rows = this.rowsOf(run, exp, key); | |
| 724 | + const result = combineResults(rows); | |
| 663 | 725 | let outputs: Record<string, string> = {}; |
| 664 | - for (const r of rows) { | |
| 665 | - if (!call) Object.assign(outputs, JSON.parse(r.outputs)); | |
| 666 | - const c = r.conclusion === "timed_out" ? "failure" : (r.conclusion ?? "failure"); | |
| 667 | - if (c === "failure") result = "failure"; | |
| 668 | - else if (c === "cancelled" && result !== "failure") result = "cancelled"; | |
| 669 | - } | |
| 670 | - if (rows.length && rows.every((r) => r.conclusion === "skipped")) result = "skipped"; | |
| 726 | + if (!call) for (const r of rows) Object.assign(outputs, JSON.parse(r.outputs)); | |
| 671 | 727 | if (call && Object.keys(call.workflow.on.workflow_call?.outputs ?? {}).length) { |
| 672 | 728 | const jobs: Record<string, NeedsResult> = {}; |
| 673 | 729 | for (const k of Object.keys(call.workflow.jobs)) jobs[k] = await this.resultOf(run, exp, `${key}/${k}`); |
| @@ -726,7 +782,6 @@ | ||
| 726 | 782 | const job = exp.workflow.jobs[jobKey]!; |
| 727 | 783 | const owner = exp.workflowOf.get(jobKey)!; |
| 728 | 784 | const parent = exp.parentOf.get(jobKey) ?? null; |
| 729 | - const cancelled = run.conclusion === "cancelled"; | |
| 730 | 785 | const chain: string[] = []; |
| 731 | 786 | for (let c = parent; c; c = exp.calls.get(c)!.parent) chain.unshift(c); |
| 732 | 787 | const prefix = chain.map((c) => exp.calls.get(c)!.job.name ?? localKey(c, exp.calls.get(c)!.parent)).join(" / "); |
| @@ -743,27 +798,23 @@ | ||
| 743 | 798 | let baseCtx: ExprContext; |
| 744 | 799 | let shouldRun: boolean; |
| 745 | 800 | try { |
| 746 | - // The callers decide first: a skipped call, or one whose needs failed, skips everything inside it. | |
| 801 | + // The callers decide first: a skipped call, or one whose ancestors failed or were skipped, skips everything inside it. | |
| 747 | 802 | for (const c of chain) { |
| 748 | 803 | const call = exp.calls.get(c)!; |
| 749 | 804 | const callNeeds = await this.needsResults(run, exp, call.needs, call.parent); |
| 750 | 805 | const outer = await this.callContext(run, exp, call.parent, false); |
| 751 | 806 | const callCtx: ExprContext = { github: this.githubContext(run, localKey(c, call.parent), ""), needs: callNeeds, inputs: outer.inputs, vars: this.variables() }; |
| 752 | - if (!evaluateIf(call.job.if, callCtx, statusFor(Object.values(callNeeds), cancelled))) return fail(label(job.name ?? short), "", "skipped"); | |
| 807 | + if (!evaluateIf(call.job.if, callCtx, this.statusOf(run, exp, c))) return fail(label(job.name ?? short), "", "skipped"); | |
| 753 | 808 | } |
| 754 | 809 | // Resolve the call's inputs and secrets now, so a missing required one fails here rather than in the runner. |
| 755 | 810 | if (parent) await this.callContext(run, exp, parent, true); |
| 756 | 811 | needs = await this.needsResults(run, exp, exp.needs.get(jobKey) ?? [], parent); |
| 757 | 812 | baseCtx = { github: this.githubContext(run, short, ""), needs, inputs: (await this.callContext(run, exp, parent, false)).inputs, vars: this.variables(), env: owner.env }; |
| 758 | - shouldRun = evaluateIf(job.if, baseCtx, statusFor(Object.values(needs), cancelled)); | |
| 813 | + shouldRun = evaluateIf(job.if, baseCtx, this.statusOf(run, exp, jobKey)); | |
| 759 | 814 | } catch (e) { |
| 760 | 815 | return fail(label(job.name ?? short), e instanceof ReusableError ? e.message : `Error evaluating 'if': ${(e as Error).message}`); |
| 761 | 816 | } |
| 762 | 817 | if (!shouldRun) return fail(label(job.name ?? short), "", "skipped"); |
| 763 | - const labels = job.runsOn.map((l) => String(evaluateValue(l, baseCtx))); | |
| 764 | - const unsupported = labels.filter((l) => !SUPPORTED_LABEL.test(l)); | |
| 765 | - const selfHosted = labels.some((l) => l.toLowerCase() === "self-hosted"); | |
| 766 | - if (unsupported.length && !selfHosted) return fail(label(job.name ?? short), `No Diffinery runner matches runs-on: ${unsupported.join(", ")}. Linux (ubuntu-*) runners are hosted; anything else needs runs-on: [self-hosted, ...] and a registered runner.`); | |
| 767 | 818 | |
| 768 | 819 | let combos: Array<Record<string, unknown>>; |
| 769 | 820 | try { |
| @@ -776,22 +827,46 @@ | ||
| 776 | 827 | const maxParallel = Number(evaluateValue(job.strategy?.maxParallel ?? combos.length, baseCtx)) || combos.length; |
| 777 | 828 | const ids: string[] = []; |
| 778 | 829 | combos.forEach((matrix, i) => { |
| 830 | + // Like GitHub, name, runs-on, timeout-minutes, continue-on-error and concurrency are evaluated for each matrix leg. | |
| 779 | 831 | const ctx = { ...baseCtx, matrix, strategy: { "fail-fast": failFast, "job-index": i, "job-total": combos.length, "max-parallel": maxParallel } }; |
| 780 | - let name = job.name ? interpolate(job.name, ctx) : short; | |
| 832 | + let name = short; | |
| 781 | 833 | if (!job.name && Object.keys(matrix).length) name = `${short} (${Object.values(matrix).map((v) => (typeof v === "object" ? JSON.stringify(v) : String(v))).join(", ")})`; |
| 782 | - name = label(name); | |
| 783 | - const timeout = Number(evaluateValue(job.timeoutMinutes, ctx)) || 360; | |
| 834 | + let leg: { labels: string[]; timeout: number; continueOnError: boolean; group: string | null; cancel: boolean }; | |
| 835 | + try { | |
| 836 | + if (job.name) name = interpolate(job.name, ctx); | |
| 837 | + const continueOnError = evaluateValue(job.continueOnError, ctx); | |
| 838 | + const cancel = job.concurrency ? evaluateValue(job.concurrency.cancelInProgress, ctx) : false; | |
| 839 | + leg = { | |
| 840 | + labels: runnerLabels(job.runsOn, ctx), | |
| 841 | + timeout: Number(evaluateValue(job.timeoutMinutes, ctx)) || 360, | |
| 842 | + continueOnError: continueOnError === true || continueOnError === "true", | |
| 843 | + group: (job.concurrency ? interpolate(job.concurrency.group, ctx).trim() : "") || null, | |
| 844 | + cancel: cancel === true || cancel === "true", | |
| 845 | + }; | |
| 846 | + } catch (e) { | |
| 847 | + fail(label(name), `Error evaluating the job for this matrix combination: ${(e as Error).message}`); | |
| 848 | + return; | |
| 849 | + } | |
| 850 | + const unsupported = leg.labels.filter((l) => !SUPPORTED_LABEL.test(l)); | |
| 851 | + const selfHosted = leg.labels.some((l) => l.toLowerCase() === "self-hosted"); | |
| 852 | + if (!leg.labels.length || (unsupported.length && !selfHosted)) { | |
| 853 | + fail(label(name), `No Diffinery runner matches runs-on: ${unsupported.join(", ") || "(no labels)"}. Linux (ubuntu-*) runners are hosted; anything else needs runs-on: [self-hosted, ...] and a registered runner.`); | |
| 854 | + return; | |
| 855 | + } | |
| 784 | 856 | const id = newId(); |
| 785 | 857 | this.sql.exec( |
| 786 | - "INSERT INTO jobs (id, run_id, job_key, name, status, matrix, strategy, timeout_minutes, labels) VALUES (?, ?, ?, ?, 'queued', ?, ?, ?, ?)", | |
| 858 | + "INSERT INTO jobs (id, run_id, job_key, name, status, matrix, strategy, timeout_minutes, labels, continue_on_error, concurrency_group, concurrency_cancel) VALUES (?, ?, ?, ?, 'queued', ?, ?, ?, ?, ?, ?, ?)", | |
| 787 | 859 | id, |
| 788 | 860 | runId, |
| 789 | 861 | jobKey, |
| 790 | - name, | |
| 862 | + label(name), | |
| 791 | 863 | JSON.stringify(matrix), |
| 792 | 864 | JSON.stringify(ctx.strategy), |
| 793 | - Math.min(timeout, 360), | |
| 794 | - selfHosted ? JSON.stringify(labels.map((l) => l.toLowerCase())) : null, | |
| 865 | + Math.min(leg.timeout, 360), | |
| 866 | + selfHosted ? JSON.stringify(leg.labels.map((l) => l.toLowerCase())) : null, | |
| 867 | + leg.continueOnError ? 1 : 0, | |
| 868 | + leg.group, | |
| 869 | + leg.cancel ? 1 : 0, | |
| 795 | 870 | ); |
| 796 | 871 | ids.push(id); |
| 797 | 872 | }); |
| @@ -806,6 +881,7 @@ | ||
| 806 | 881 | |
| 807 | 882 | private async launch(jobId: string): Promise<void> { |
| 808 | 883 | const job = this.jobRow(jobId); |
| 884 | + if (job.status === "completed" || !(await this.concurrencySlot(job))) return; | |
| 809 | 885 | const run = this.runRow(job.run_id); |
| 810 | 886 | const m = this.repo.meta; |
| 811 | 887 | if (job.labels) { |
| @@ -826,6 +902,33 @@ | ||
| 826 | 902 | } |
| 827 | 903 | } |
| 828 | 904 | |
| 905 | + /** | |
| 906 | + * Job-level `concurrency`, like GitHub: at most one job of a group runs at a time in | |
| 907 | + * the repository. With cancel-in-progress the newcomer cancels the group's other jobs; | |
| 908 | + * without it the newcomer waits as `pending` and replaces any older pending job. | |
| 909 | + * Returns whether the job may start now. | |
| 910 | + */ | |
| 911 | + private async concurrencySlot(job: JobRow): Promise<boolean> { | |
| 912 | + if (!job.concurrency_group) return true; | |
| 913 | + const others = this.sql.exec<JobRow>("SELECT * FROM jobs WHERE concurrency_group = ? AND id != ? AND status IN ('pending', 'waiting', 'in_progress') ORDER BY rowid", job.concurrency_group, job.id).toArray(); | |
| 914 | + const pending = others.filter((o) => o.status === "pending"); | |
| 915 | + const running = others.filter((o) => o.status !== "pending"); | |
| 916 | + const failure = { message: `Canceling since a higher priority waiting request for '${job.concurrency_group}' exists` }; | |
| 917 | + if (running.length && !job.concurrency_cancel) { | |
| 918 | + this.sql.exec("UPDATE jobs SET status = 'pending' WHERE id = ?", job.id); | |
| 919 | + this.repo.broadcast(`run:${job.run_id}`, { action: "job", jobId: job.id }); | |
| 920 | + for (const o of pending) await this.completeJob(o.id, { conclusion: "cancelled", outputs: {}, failure }); | |
| 921 | + return false; | |
| 922 | + } | |
| 923 | + // Take the group before cancelling anything, so a job that starts meanwhile waits for this one. | |
| 924 | + this.sql.exec("UPDATE jobs SET status = 'waiting' WHERE id = ?", job.id); | |
| 925 | + for (const o of [...pending, ...running]) { | |
| 926 | + await this.stopContainer(o); | |
| 927 | + await this.completeJob(o.id, { conclusion: "cancelled", outputs: {}, failure }); | |
| 928 | + } | |
| 929 | + return true; | |
| 930 | + } | |
| 931 | + | |
| 829 | 932 | async jobTimedOut(jobId: string): Promise<void> { |
| 830 | 933 | const j = this.jobRow(jobId); |
| 831 | 934 | if (j.status === "completed") return; |
| @@ -870,10 +973,11 @@ | ||
| 870 | 973 | |
| 871 | 974 | async finishRun(runId: string): Promise<RunInfo> { |
| 872 | 975 | const run = this.runRow(runId); |
| 873 | - const jobs = this.jobs(runId); | |
| 976 | + const jobs = this.sql.exec<JobRow>("SELECT * FROM jobs WHERE run_id = ?", runId).toArray(); | |
| 874 | 977 | let conclusion: RunRow["conclusion"] = run.conclusion === "cancelled" ? "cancelled" : "success"; |
| 875 | 978 | if (conclusion !== "cancelled") { |
| 876 | - if (jobs.some((j) => j.conclusion === "failure" || j.conclusion === "timed_out")) conclusion = "failure"; | |
| 979 | + // A job with continue-on-error may fail without failing the run. | |
| 980 | + if (jobs.some((j) => (j.conclusion === "failure" || j.conclusion === "timed_out") && !j.continue_on_error)) conclusion = "failure"; | |
| 877 | 981 | else if (jobs.some((j) => j.conclusion === "cancelled")) conclusion = "cancelled"; |
| 878 | 982 | } |
| 879 | 983 | const t = now(); |
| @@ -903,7 +1007,7 @@ | ||
| 903 | 1007 | const branch = payload.pull_request?.head?.ref ?? run.ref.replace(/^refs\/(heads|tags)\//, ""); |
| 904 | 1008 | const actor = JSON.parse(run.actor) as Actor; |
| 905 | 1009 | const workflowRun = { |
| 906 | - id: run.id, | |
| 1010 | + id: run.run_number, | |
| 907 | 1011 | name: run.workflow_name, |
| 908 | 1012 | display_title: run.display_title, |
| 909 | 1013 | path: run.workflow_path, |
| @@ -927,7 +1031,7 @@ | ||
| 927 | 1031 | event: "workflow_run", |
| 928 | 1032 | ref: this.repo.headRef, |
| 929 | 1033 | sha, |
| 930 | - payload: { action, workflow_run: workflowRun, workflow: { name: run.workflow_name, path: run.workflow_path }, repository: this.repositoryPayload(), sender: { login: actor.login } }, | |
| 1034 | + payload: { action, workflow_run: workflowRun, workflow: { name: run.workflow_name, path: run.workflow_path }, repository: await this.repositoryPayload(), sender: { login: actor.login } }, | |
| 931 | 1035 | caller: actor, |
| 932 | 1036 | title: w.workflow!.name ?? w.path, |
| 933 | 1037 | }); |
| @@ -941,8 +1045,8 @@ | ||
| 941 | 1045 | private chainDepth(run: RunRow): number { |
| 942 | 1046 | let depth = 0; |
| 943 | 1047 | for (let r: RunRow | undefined = run; r?.event === "workflow_run" && depth < 3; depth++) { |
| 944 | - const parent: string | undefined = (JSON.parse(r.event_payload) as { workflow_run?: { id?: string } }).workflow_run?.id; | |
| 945 | - r = parent ? this.sql.exec<RunRow>("SELECT * FROM runs WHERE id = ?", parent).toArray()[0] : undefined; | |
| 1048 | + const parent: string | number | undefined = (JSON.parse(r.event_payload) as { workflow_run?: { id?: string | number } }).workflow_run?.id; | |
| 1049 | + r = parent !== undefined ? this.sql.exec<RunRow>("SELECT * FROM runs WHERE id = ? OR run_number = ? ORDER BY attempt DESC LIMIT 1", String(parent), Number(parent) || -1).toArray()[0] : undefined; | |
| 946 | 1050 | } |
| 947 | 1051 | return depth; |
| 948 | 1052 | } |
| @@ -957,8 +1061,8 @@ | ||
| 957 | 1061 | private async cancelInternal(runId: string, reason: string): Promise<void> { |
| 958 | 1062 | const run = this.runRow(runId); |
| 959 | 1063 | if (run.status === "completed") return; |
| 960 | - this.sql.exec("UPDATE runs SET conclusion = 'cancelled', updated_at = ? WHERE id = ?", now(), runId); | |
| 961 | - for (const j of this.sql.exec<JobRow>("SELECT * FROM jobs WHERE run_id = ? AND status != 'completed'", runId).toArray()) { | |
| 1064 | + this.sql.exec("UPDATE runs SET conclusion = 'cancelled', updated_at = ? WHERE id = ?", now(), run.id); | |
| 1065 | + for (const j of this.sql.exec<JobRow>("SELECT * FROM jobs WHERE run_id = ? AND status != 'completed'", run.id).toArray()) { | |
| 962 | 1066 | await this.stopContainer(j); |
| 963 | 1067 | await this.completeJob(j.id, { conclusion: "cancelled", outputs: {}, failure: { message: reason } }); |
| 964 | 1068 | } |
| @@ -971,6 +1075,7 @@ | ||
| 971 | 1075 | |
| 972 | 1076 | async rerun(runId: string, caller: Caller): Promise<RunInfo> { |
| 973 | 1077 | if (!atLeast(caller.role, "write")) throw new HttpError(403, "permission denied"); |
| 1078 | + if (caller.id.startsWith("job:")) throw new HttpError(403, "a workflow's GITHUB_TOKEN cannot re-run workflows"); | |
| 974 | 1079 | const old = this.runRow(runId); |
| 975 | 1080 | const wf = (await this.loadWorkflows(old.sha)).find((w) => w.path === old.workflow_path)?.workflow; |
| 976 | 1081 | if (!wf) throw new HttpError(422, "workflow file no longer valid at this commit"); |
| @@ -984,6 +1089,8 @@ | ||
| 984 | 1089 | prNumber: old.pr_number ?? undefined, |
| 985 | 1090 | inputs: JSON.parse(old.inputs), |
| 986 | 1091 | title: old.display_title, |
| 1092 | + // Re-running code an agent wrote keeps it acting as the agent, whoever re-runs it. | |
| 1093 | + agentOrigin: !!old.agent_origin, | |
| 987 | 1094 | }); |
| 988 | 1095 | this.sql.exec("UPDATE runs SET attempt = ?, run_number = ? WHERE id = ?", old.attempt + 1, old.run_number, run!.id); |
| 989 | 1096 | return toRun(this.runRow(run!.id)); |
| @@ -992,14 +1099,20 @@ | ||
| 992 | 1099 | // ---------------------------------------------------------------- runner protocol |
| 993 | 1100 | |
| 994 | 1101 | /** Validate a job or sandbox token. Returns what it grants. */ |
| 995 | - async runnerAuth(token: string): Promise<{ kind: "job"; jobId: string; runId: string } | { kind: "sandbox"; sandboxId: string; caller: Caller } | null> { | |
| 1102 | + async runnerAuth( | |
| 1103 | + token: string, | |
| 1104 | + ): Promise<{ kind: "job"; jobId: string; runId: string; role: Caller["role"]; permissions: JobPermissions; actorKind: Actor["kind"] } | { kind: "sandbox"; sandboxId: string; caller: Caller } | null> { | |
| 996 | 1105 | const parts = token.split("_"); |
| 997 | 1106 | if (parts.length < 4) return null; |
| 998 | 1107 | const hash = await sha256(token); |
| 999 | 1108 | if (parts[0] === "djt") { |
| 1000 | 1109 | const j = this.sql.exec<JobRow>("SELECT * FROM jobs WHERE id = ?", parts[2]!).toArray()[0]; |
| 1001 | 1110 | if (!j || j.token_hash !== hash || j.status === "completed") return null; |
| 1002 | - return { kind: "job", jobId: j.id, runId: j.run_id }; | |
| 1111 | + const grant = this.repo.runtime.grantFor(j.id); | |
| 1112 | + // Code an agent wrote, pushed or opened a pull request with runs as that agent (repo/token-policy.ts). | |
| 1113 | + const run = this.sql.exec<{ agent_origin: number }>("SELECT agent_origin FROM runs WHERE id = ?", j.run_id).toArray()[0]; | |
| 1114 | + const actorKind: Actor["kind"] = run?.agent_origin ? "agent" : "system"; | |
| 1115 | + return { kind: "job", jobId: j.id, runId: j.run_id, role: grant.role, permissions: grant.permissions, actorKind }; | |
| 1003 | 1116 | } |
| 1004 | 1117 | if (parts[0] === "dst") { |
| 1005 | 1118 | const s = this.sql.exec<{ id: string; secret: string; created_by: string; status: string }>("SELECT * FROM sandboxes WHERE id = ?", parts[2]!).toArray()[0]; |
| @@ -1023,7 +1136,7 @@ | ||
| 1023 | 1136 | const exp = await this.expanded(run); |
| 1024 | 1137 | const def: Job = exp.workflow.jobs[job.job_key]!; |
| 1025 | 1138 | const wf = exp.workflowOf.get(job.job_key)!; |
| 1026 | - const allowed = wf.permissions === "write-all" || (typeof wf.permissions === "object" && wf.permissions["id-token"] === "write") || (typeof def.permissions === "object" && def.permissions["id-token"] === "write") || def.permissions === "write-all"; | |
| 1139 | + const allowed = this.repo.runtime.grantFor(jobId).permissions["id-token"] === "write"; | |
| 1027 | 1140 | const ctx = this.githubContext(run, localKey(job.job_key, exp.parentOf.get(job.job_key) ?? null), token) as Record<string, Json>; |
| 1028 | 1141 | const github: Record<string, string> = {}; |
| 1029 | 1142 | for (const [k, v] of Object.entries(ctx)) if (typeof v === "string" || typeof v === "number") github[k] = String(v); |
| @@ -1037,16 +1150,17 @@ | ||
| 1037 | 1150 | */ |
| 1038 | 1151 | async packagesPermission(token: string): Promise<"none" | "read" | "write"> { |
| 1039 | 1152 | const auth = await this.runnerAuth(token); |
| 1040 | - if (!auth || auth.kind !== "job") return "none"; | |
| 1041 | - const job = this.jobRow(auth.jobId); | |
| 1042 | - const exp = await this.expanded(this.runRow(job.run_id)); | |
| 1043 | - const perms = exp.workflow.jobs[job.job_key]?.permissions ?? exp.workflowOf.get(job.job_key)?.permissions; | |
| 1044 | - if (perms === undefined || perms === "write-all") return "write"; | |
| 1045 | - if (perms === "read-all") return "read"; | |
| 1046 | - const v = perms.packages; | |
| 1047 | - return v === "write" || v === "read" ? v : "none"; | |
| 1153 | + return auth?.kind === "job" ? auth.permissions.packages : "none"; | |
| 1048 | 1154 | } |
| 1049 | 1155 | |
| 1156 | + /** | |
| 1157 | + * A pull_request run for a pull request from another repository (a fork): its jobs | |
| 1158 | + * never get write. Decided from the pull request's row, not the event payload. | |
| 1159 | + */ | |
| 1160 | + private untrusted(run: RunRow): boolean { | |
| 1161 | + return run.event === "pull_request" && this.repo.policy.isForkPull(run.pr_number); | |
| 1162 | + } | |
| 1163 | + | |
| 1050 | 1164 | async runnerSpec(jobId: string, token: string): Promise<JobSpec> { |
| 1051 | 1165 | const job = await this.authJob(jobId, token); |
| 1052 | 1166 | const run = this.runRow(job.run_id); |
| @@ -1067,11 +1181,13 @@ | ||
| 1067 | 1181 | let environment: { secrets: Record<string, string>; vars: Record<string, string> } = { secrets: {}, vars: {} }; |
| 1068 | 1182 | if (def.environment) { |
| 1069 | 1183 | try { |
| 1070 | - const ctx = { github: this.githubContext(run, short, ""), inputs: scope.inputs, vars: this.variables(), matrix: JSON.parse(job.matrix) }; | |
| 1184 | + const ctx = { github: this.githubContext(run, short, ""), inputs: scope.inputs, vars: this.variables(), matrix: JSON.parse(job.matrix), strategy: JSON.parse(job.strategy), needs: await this.needsResults(run, exp, exp.needs.get(job.job_key) ?? [], parent) }; | |
| 1071 | 1185 | const name = interpolate(def.environment.name, ctx); |
| 1072 | 1186 | const url = def.environment.url && !def.environment.url.includes("steps.") ? interpolate(def.environment.url, ctx) : undefined; |
| 1073 | 1187 | const logUrl = `${this.env.PUBLIC_URL}/${m.owner}/${m.name}/actions/runs/${run.id}/job/${jobId}`; |
| 1074 | - environment = await this.repo.deploys.jobStarted({ id: jobId, ref: run.ref, sha: run.sha, logUrl, actor: JSON.parse(run.actor) as Actor }, { name, url }); | |
| 1188 | + // Code an agent wrote deploys as that agent (environment rules hold it to the agent's limits). | |
| 1189 | + const actor = JSON.parse(run.actor) as Actor; | |
| 1190 | + environment = await this.repo.deploys.jobStarted({ id: jobId, ref: run.ref, sha: run.sha, logUrl, actor: run.agent_origin ? { ...actor, kind: "agent" } : actor }, { name, url }); | |
| 1075 | 1191 | } catch (e) { |
| 1076 | 1192 | await this.completeJob(jobId, { conclusion: "failure", outputs: {}, failure: { message: (e as Error).message } }); |
| 1077 | 1193 | throw new HttpError(409, (e as Error).message); |
| @@ -1082,7 +1198,10 @@ | ||
| 1082 | 1198 | this.repo.upsertCheck({ sha: run.head_sha ?? run.sha, name: this.checkName(run, job), status: "in_progress", runId: run.id, jobId, detailsUrl: `${this.env.PUBLIC_URL}/${m.owner}/${m.name}/actions/runs/${run.id}/job/${jobId}` }); |
| 1083 | 1199 | this.repo.broadcast(`run:${run.id}`, { action: "job", jobId }); |
| 1084 | 1200 | const secrets: Record<string, string> = { ...scope.secrets, ...environment.secrets, GITHUB_TOKEN: token, DIFFINERY_TOKEN: token }; |
| 1085 | - const idToken = wf.permissions === "write-all" || (typeof wf.permissions === "object" && wf.permissions["id-token"] === "write") || (typeof def.permissions === "object" && def.permissions["id-token"] === "write") || def.permissions === "write-all"; | |
| 1201 | + // The job token can do what the effective `permissions:` allow (job level wins, like GitHub). | |
| 1202 | + const permissions = permissionsForJob(exp, job.job_key, { untrusted: this.untrusted(run), repoDefault: this.repo.policy.settings().defaultPermissions }); | |
| 1203 | + this.repo.runtime.grant(jobId, permissions); | |
| 1204 | + const idToken = permissions["id-token"] === "write"; | |
| 1086 | 1205 | return { |
| 1087 | 1206 | runId: run.id, |
| 1088 | 1207 | jobId, |
| @@ -1106,13 +1225,24 @@ | ||
| 1106 | 1225 | container: def.container, |
| 1107 | 1226 | services: def.services, |
| 1108 | 1227 | timeoutMinutes: job.timeout_minutes, |
| 1109 | - continueOnError: def.continueOnError === true || def.continueOnError === "true", | |
| 1228 | + continueOnError: !!job.continue_on_error, | |
| 1110 | 1229 | idToken: !!idToken, |
| 1111 | 1230 | apiUrl: this.env.PUBLIC_URL, |
| 1112 | 1231 | token, |
| 1232 | + runtime: await this.runtimeServices(run, job), | |
| 1113 | 1233 | }; |
| 1114 | 1234 | } |
| 1115 | 1235 | |
| 1236 | + /** ACTIONS_RUNTIME_TOKEN and the service URLs; the token expires when the job would time out. */ | |
| 1237 | + private async runtimeServices(run: RunRow, job: JobRow): Promise<JobSpec["runtime"]> { | |
| 1238 | + if (!this.env.MASTER_KEY) return undefined; | |
| 1239 | + const origin = new URL(this.env.PUBLIC_URL).origin; | |
| 1240 | + const exp = Math.floor(Date.now() / 1000) + (job.timeout_minutes + 30) * 60; | |
| 1241 | + const { scopes } = this.repo.runtime.job(job.id); | |
| 1242 | + const token = await signRuntimeToken(this.env.MASTER_KEY, origin, { repo: this.repo.meta.repoId, run: run.id, job: job.id, exp }, scopes); | |
| 1243 | + return { token, resultsUrl: `${origin}/`, cacheUrl: `${origin}/_runtime/`, runtimeUrl: `${origin}/_runtime/` }; | |
| 1244 | + } | |
| 1245 | + | |
| 1116 | 1246 | async runnerLogs(jobId: string, token: string, lines: LogLine[]): Promise<void> { |
| 1117 | 1247 | await this.authJob(jobId, token); |
| 1118 | 1248 | let seq = this.sql.exec<{ n: number | null }>("SELECT max(seq) AS n FROM logs WHERE job_id = ?", jobId).one().n ?? -1; |
| @@ -1178,13 +1308,15 @@ | ||
| 1178 | 1308 | await this.env.BUCKET.put(this.logKey(jobId), JSON.stringify(lines), { httpMetadata: { contentType: "application/json" } }); |
| 1179 | 1309 | this.sql.exec("DELETE FROM logs WHERE job_id = ?", jobId); |
| 1180 | 1310 | } |
| 1311 | + // Uploads the job left unfinished stop counting against the budgets, and their bytes go. | |
| 1312 | + await this.repo.runtime.sweepJob(jobId).catch((e) => console.error("runtime sweep failed", e)); | |
| 1181 | 1313 | this.repo.broadcast(`job:${jobId}`, { action: "completed", conclusion }); |
| 1182 | 1314 | this.repo.broadcast(`run:${run.id}`, { action: "job", jobId }); |
| 1183 | 1315 | |
| 1184 | - // fail-fast: cancel unfinished siblings of a failed matrix job. | |
| 1316 | + // fail-fast: cancel unfinished siblings of a failed matrix job (unless the failed leg may continue on error). | |
| 1185 | 1317 | const siblings = this.sql.exec<JobRow>("SELECT * FROM jobs WHERE run_id = ? AND job_key = ? AND id != ? AND status != 'completed'", run.id, job.job_key, jobId).toArray(); |
| 1186 | 1318 | const strategy = JSON.parse(job.strategy || "{}") as { "fail-fast"?: boolean }; |
| 1187 | - if (conclusion === "failure" && strategy["fail-fast"] !== false && siblings.length) { | |
| 1319 | + if ((conclusion === "failure" || conclusion === "timed_out") && !job.continue_on_error && strategy["fail-fast"] !== false && siblings.length) { | |
| 1188 | 1320 | for (const s of siblings) { |
| 1189 | 1321 | await this.stopContainer(s); |
| 1190 | 1322 | await this.completeJob(s.id, { conclusion: "cancelled", outputs: {}, failure: { message: `Cancelled because ${job.name} failed (fail-fast).` } }); |
| @@ -1193,6 +1325,11 @@ | ||
| 1193 | 1325 | const queued = siblings.find((s) => s.status === "queued"); |
| 1194 | 1326 | if (queued) await this.launch(queued.id); |
| 1195 | 1327 | } |
| 1328 | + // A job that held its concurrency group hands it to the oldest pending job (of any run). | |
| 1329 | + if (job.concurrency_group && (job.status === "waiting" || job.status === "in_progress")) { | |
| 1330 | + const next = this.sql.exec<{ id: string }>("SELECT id FROM jobs WHERE concurrency_group = ? AND status = 'pending' ORDER BY rowid LIMIT 1", job.concurrency_group).toArray()[0]; | |
| 1331 | + if (next) await this.launch(next.id); | |
| 1332 | + } | |
| 1196 | 1333 | try { |
| 1197 | 1334 | const instance = await this.env.RUN_WORKFLOW.get(run.id); |
| 1198 | 1335 | await instance.sendEvent({ type: `job-${jobId}`, payload: { conclusion } }); |
| @@ -1244,38 +1381,75 @@ | ||
| 1244 | 1381 | |
| 1245 | 1382 | // ---------------------------------------------------------------- cache and artifacts |
| 1246 | 1383 | |
| 1247 | - async cacheLookup(jobId: string, token: string, key: string, restoreKeys: string[]): Promise<{ key: string; r2Key: string } | null> { | |
| 1384 | + /** | |
| 1385 | + * An older runner's cache (the runner API before the toolkit's cache service). | |
| 1386 | + * Entries are stored as `<ref>\n<key>` and scoped like the cache service's | |
| 1387 | + * (repo/runtime.ts): a job saves to its own ref and restores from its own ref, | |
| 1388 | + * then the pull request's base, then the default branch, so one branch or pull | |
| 1389 | + * request never sees another's caches. Entries saved before scoping (no ref) | |
| 1390 | + * are never restored and age out of the budget. | |
| 1391 | + */ | |
| 1392 | + async cacheLookup(jobId: string, token: string, key: string, restoreKeys: string[]): Promise<{ key: string; stored: string } | null> { | |
| 1248 | 1393 | await this.authJob(jobId, token); |
| 1249 | - const exact = this.sql.exec<{ key: string }>("SELECT key FROM caches WHERE key = ?", key).toArray()[0]; | |
| 1250 | - let hit = exact?.key; | |
| 1251 | - if (!hit) { | |
| 1394 | + const { scopes } = this.repo.runtime.job(jobId); | |
| 1395 | + // An entry whose object is gone is forgotten, and the lookup moves on. | |
| 1396 | + for (let attempt = 0; attempt < 5; attempt++) { | |
| 1397 | + const hit = this.legacyCacheMatch(scopes, key, restoreKeys); | |
| 1398 | + if (!hit) return null; | |
| 1399 | + if (!(await headFirst(this.env.BUCKET, this.cacheR2Keys(hit)))) { | |
| 1400 | + this.sql.exec("DELETE FROM caches WHERE key = ?", hit); | |
| 1401 | + continue; | |
| 1402 | + } | |
| 1403 | + this.sql.exec("UPDATE caches SET last_used = ? WHERE key = ?", now(), hit); | |
| 1404 | + return { key: hit.slice(hit.indexOf("\n") + 1), stored: hit }; | |
| 1405 | + } | |
| 1406 | + return null; | |
| 1407 | + } | |
| 1408 | + | |
| 1409 | + /** In each scope in turn: the exact key, then the newest entry for each restore key prefix. */ | |
| 1410 | + private legacyCacheMatch(scopes: string[], key: string, restoreKeys: string[]): string | null { | |
| 1411 | + for (const scope of scopes) { | |
| 1412 | + const exact = this.sql.exec<{ key: string }>("SELECT key FROM caches WHERE key = ?", `${scope}\n${key}`).toArray()[0]; | |
| 1413 | + if (exact) return exact.key; | |
| 1252 | 1414 | for (const prefix of restoreKeys) { |
| 1253 | - const r = this.sql.exec<{ key: string }>("SELECT key FROM caches WHERE key >= ? AND key < ? ORDER BY created_at DESC LIMIT 1", prefix, `${prefix}`).toArray()[0]; | |
| 1254 | - if (r) { | |
| 1255 | - hit = r.key; | |
| 1256 | - break; | |
| 1257 | - } | |
| 1415 | + const p = `${scope}\n${prefix}`; | |
| 1416 | + const r = this.sql.exec<{ key: string }>("SELECT key FROM caches WHERE substr(key, 1, ?) = ? ORDER BY created_at DESC LIMIT 1", p.length, p).toArray()[0]; | |
| 1417 | + if (r) return r.key; | |
| 1258 | 1418 | } |
| 1259 | 1419 | } |
| 1260 | - if (!hit) return null; | |
| 1261 | - this.sql.exec("UPDATE caches SET last_used = ? WHERE key = ?", now(), hit); | |
| 1262 | - return { key: hit, r2Key: this.cacheR2Key(hit) }; | |
| 1420 | + return null; | |
| 1263 | 1421 | } |
| 1264 | 1422 | |
| 1265 | - cacheR2Key(key: string): string { | |
| 1266 | - return `repos/${this.repo.meta.repoId}/cache/${encodeURIComponent(key)}`; | |
| 1423 | + /** Where a job saves `key` (its own ref's scope): the stored key and its R2 keys. */ | |
| 1424 | + async cacheSaveKeys(jobId: string, token: string, key: string): Promise<{ stored: string; r2Keys: string[] }> { | |
| 1425 | + await this.authJob(jobId, token); | |
| 1426 | + const stored = `${this.repo.runtime.job(jobId).scopes[0]}\n${key}`; | |
| 1427 | + return { stored, r2Keys: this.cacheR2Keys(stored) }; | |
| 1267 | 1428 | } |
| 1268 | 1429 | |
| 1269 | - async cacheSaved(jobId: string, token: string, key: string, size: number): Promise<string[]> { | |
| 1430 | + /** The R2 keys of entry `stored`, when the job may read its scope. */ | |
| 1431 | + async cacheReadKeys(jobId: string, token: string, stored: string): Promise<string[] | null> { | |
| 1270 | 1432 | await this.authJob(jobId, token); |
| 1271 | - this.sql.exec("INSERT OR REPLACE INTO caches (key, size, created_at, last_used) VALUES (?, ?, ?, ?)", key, size, now(), now()); | |
| 1433 | + const nl = stored.indexOf("\n"); | |
| 1434 | + if (nl < 0 || !this.repo.runtime.job(jobId).scopes.includes(stored.slice(0, nl))) return null; | |
| 1435 | + return this.cacheR2Keys(stored); | |
| 1436 | + } | |
| 1437 | + | |
| 1438 | + /** Where an older runner's cache entry is stored: the current key, then the old layout's. */ | |
| 1439 | + cacheR2Keys(key: string): string[] { | |
| 1440 | + return legacyCacheKeys(this.repo.meta.repoId, key); | |
| 1441 | + } | |
| 1442 | + | |
| 1443 | + async cacheSaved(jobId: string, token: string, key: string, size: number): Promise<string[]> { | |
| 1444 | + const { stored } = await this.cacheSaveKeys(jobId, token, key); | |
| 1445 | + this.sql.exec("INSERT OR REPLACE INTO caches (key, size, created_at, last_used) VALUES (?, ?, ?, ?)", stored, size, now(), now()); | |
| 1272 | 1446 | // Evict least recently used entries above 10 GB per repository. |
| 1273 | 1447 | const evicted: string[] = []; |
| 1274 | 1448 | let total = this.sql.exec<{ n: number | null }>("SELECT sum(size) AS n FROM caches").one().n ?? 0; |
| 1275 | 1449 | while (total > 10 * 1024 ** 3) { |
| 1276 | 1450 | const old = this.sql.exec<{ key: string; size: number }>("SELECT key, size FROM caches ORDER BY last_used LIMIT 1").one(); |
| 1277 | 1451 | this.sql.exec("DELETE FROM caches WHERE key = ?", old.key); |
| 1278 | - evicted.push(this.cacheR2Key(old.key)); | |
| 1452 | + evicted.push(...this.cacheR2Keys(old.key)); | |
| 1279 | 1453 | total -= old.size; |
| 1280 | 1454 | } |
| 1281 | 1455 | return evicted; |
| @@ -1284,15 +1458,34 @@ | ||
| 1284 | 1458 | async artifactSaved(jobId: string, token: string, name: string, size: number): Promise<{ r2Key: string }> { |
| 1285 | 1459 | const job = await this.authJob(jobId, token); |
| 1286 | 1460 | this.sql.exec("INSERT OR REPLACE INTO artifacts (id, run_id, name, size, created_at) VALUES (?, ?, ?, ?, ?)", newId(), job.run_id, name, size, now()); |
| 1287 | - return { r2Key: this.artifactR2Key(job.run_id, name) }; | |
| 1461 | + return { r2Key: this.artifactR2Keys(job.run_id, name)[0]! }; | |
| 1288 | 1462 | } |
| 1289 | 1463 | |
| 1290 | - artifactR2Key(runId: string, name: string): string { | |
| 1291 | - return `repos/${this.repo.meta.repoId}/artifacts/${runId}/${encodeURIComponent(name)}`; | |
| 1464 | + /** Where an older runner's artifact is stored: the current key, then the old layout's. */ | |
| 1465 | + artifactR2Keys(runId: string, name: string): string[] { | |
| 1466 | + return legacyArtifactKeys(this.repo.meta.repoId, runId, name); | |
| 1292 | 1467 | } |
| 1293 | 1468 | |
| 1294 | - artifacts(runId: string): Array<{ name: string; size: number; createdAt: string; r2Key: string }> { | |
| 1295 | - return this.sql.exec<{ name: string; size: number; created_at: string }>("SELECT name, size, created_at FROM artifacts WHERE run_id = ? ORDER BY name", runId).toArray().map((a) => ({ name: a.name, size: a.size, createdAt: a.created_at, r2Key: this.artifactR2Key(runId, a.name) })); | |
| 1469 | + /** | |
| 1470 | + * A run's artifacts: those uploaded through the artifact service (zip files, or | |
| 1471 | + * single unarchived files) and older runners' tarballs. | |
| 1472 | + */ | |
| 1473 | + artifacts(runId: string): Array<{ name: string; size: number; createdAt: string; r2Keys: string[]; id?: number; contentType: string; fileName: string }> { | |
| 1474 | + runId = this.runRow(runId).id; | |
| 1475 | + const legacy = this.sql | |
| 1476 | + .exec<{ name: string; size: number; created_at: string }>("SELECT name, size, created_at FROM artifacts WHERE run_id = ? ORDER BY name", runId) | |
| 1477 | + .toArray() | |
| 1478 | + .map((a) => ({ name: a.name, size: a.size, createdAt: a.created_at, r2Keys: this.artifactR2Keys(runId, a.name), contentType: "application/gzip", fileName: `${a.name}.tar.gz` })); | |
| 1479 | + const current = this.repo.runtime.runArtifacts(runId).map((a) => ({ | |
| 1480 | + name: a.name, | |
| 1481 | + size: a.size, | |
| 1482 | + createdAt: a.createdAt, | |
| 1483 | + r2Keys: [finalKey({ repo: this.repo.meta.repoId, kind: "artifact", id: a.id })], | |
| 1484 | + id: a.id, | |
| 1485 | + contentType: a.mimeType, | |
| 1486 | + fileName: a.mimeType === "application/zip" ? `${a.name}.zip` : a.name, | |
| 1487 | + })); | |
| 1488 | + return [...current, ...legacy.filter((l) => !current.some((c) => c.name === l.name))].sort((a, b) => a.name.localeCompare(b.name)); | |
| 1296 | 1489 | } |
| 1297 | 1490 | |
| 1298 | 1491 | /** Runs of the same commit; lets a job download artifacts uploaded by earlier jobs. */ |
modifiedapps/diffinery/src/repo/collab.ts+26−7View file
| @@ -131,6 +131,9 @@ | ||
| 131 | 131 | mergeBase: string | null; |
| 132 | 132 | /** Set when the base branch merges through a merge queue. */ |
| 133 | 133 | mergeQueue: { method: "merge" | "squash" | "rebase"; entry: QueueEntry | null } | null; |
| 134 | + /** The base branch's rules that decide the merge form: merge commits refused, admins held to the rules. */ | |
| 135 | + linearHistory?: boolean; | |
| 136 | + enforceAdmins?: boolean; | |
| 134 | 137 | }; |
| 135 | 138 | |
| 136 | 139 | const actorOf = (c: Caller) => ({ id: c.id, login: c.login, kind: c.kind, onBehalfOf: c.onBehalfOf }); |
| @@ -318,6 +321,8 @@ | ||
| 318 | 321 | |
| 319 | 322 | createIssue(input: { title: string; body?: string; labels?: string[]; assignees?: string[] }, caller: Caller): Issue { |
| 320 | 323 | if (!input.title?.trim()) throw new HttpError(422, "title is required"); |
| 324 | + // As on GitHub, labels and assignees from someone without triage access are silently dropped. | |
| 325 | + if (!atLeast(caller.role, "triage")) input = { ...input, labels: [], assignees: [] }; | |
| 321 | 326 | const n = this.nextNumber(); |
| 322 | 327 | const t = now(); |
| 323 | 328 | this.sql.exec( |
| @@ -400,7 +405,8 @@ | ||
| 400 | 405 | this.sql.exec("UPDATE pulls SET draft = ? WHERE number = ?", draft ? 1 : 0, number); |
| 401 | 406 | this.sql.exec("UPDATE issues SET updated_at = ? WHERE number = ?", t, number); |
| 402 | 407 | const action = draft ? "converted_to_draft" : "ready_for_review"; |
| 403 | - this.repo.social.record(number, action, caller, {}); | |
| 408 | + // GitHub names the timeline event convert_to_draft; the webhook action is converted_to_draft. | |
| 409 | + this.repo.social.record(number, draft ? "convert_to_draft" : "ready_for_review", caller, {}); | |
| 404 | 410 | const m = this.repo.meta; |
| 405 | 411 | this.repo.emit({ type: "pull_request", action, repoId: m.repoId, owner: m.owner, repo: m.name, number, headRef: p.head_ref, headSha: p.head_sha, baseRef: p.base_ref, actor: actorOf(caller), at: t }); |
| 406 | 412 | this.repo.background(this.repo.actions.onPullRequest(action, number, caller).catch((e) => console.error("pr workflow trigger", e))); |
| @@ -442,6 +448,7 @@ | ||
| 442 | 448 | // ---------------------------------------------------------------- pull requests |
| 443 | 449 | |
| 444 | 450 | async createPull(input: { title: string; body?: string; head: string; base?: string; draft?: boolean; intent?: string; reviewers?: string[] }, caller: Caller): Promise<Issue> { |
| 451 | + this.repo.policy.assertMayCreateOrApprove(caller, "create"); | |
| 445 | 452 | const base = input.base ?? this.repo.meta.defaultBranch; |
| 446 | 453 | if (input.head === base) throw new HttpError(422, "head and base must be different branches"); |
| 447 | 454 | const headSha = this.repo.getRef(`refs/heads/${input.head}`); |
| @@ -574,6 +581,8 @@ | ||
| 574 | 581 | if (r.locked && !atLeast(caller.role, "triage")) throw new HttpError(403, "conversation is locked"); |
| 575 | 582 | if (input.event !== "COMMENT" && r.author_id === caller.id) throw new HttpError(422, "you cannot approve or request changes on your own pull request"); |
| 576 | 583 | if (input.event !== "COMMENT" && !atLeast(caller.role, "write")) throw new HttpError(403, "write access is required to approve or request changes"); |
| 584 | + // A workflow's GITHUB_TOKEN approves (or blocks) only where the repository allows it, as on GitHub. | |
| 585 | + if (input.event !== "COMMENT") this.repo.policy.assertMayCreateOrApprove(caller, "approve"); | |
| 577 | 586 | if (input.event === "COMMENT" && !input.body?.trim() && !input.comments?.length) throw new HttpError(422, "a comment review needs a body or line comments"); |
| 578 | 587 | const id = newId(); |
| 579 | 588 | const t = now(); |
| @@ -700,12 +709,15 @@ | ||
| 700 | 709 | result.conflicts = merged.conflicts; |
| 701 | 710 | const rules = this.repo.protectionFor(p.base_ref) ?? {}; |
| 702 | 711 | if (rules.mergeQueue) result.mergeQueue = { method: rules.mergeQueue.method ?? "merge", entry: this.repo.queue.entry(number) }; |
| 712 | + result.linearHistory = !!rules.requireLinearHistory; | |
| 713 | + result.enforceAdmins = !!rules.enforceAdmins; | |
| 703 | 714 | result.requiredApprovals = rules.requiredApprovals ?? 0; |
| 704 | 715 | const latest = new Map<string, Review>(); |
| 705 | 716 | for (const rv of this.reviews(number)) if (rv.state !== "COMMENTED") latest.set(rv.author.id, rv); |
| 706 | 717 | for (const rv of latest.values()) { |
| 707 | - // Agent reviews are advisory: only people's approvals count toward requirements. | |
| 708 | - if (rv.state === "APPROVED" && rv.author.kind !== "agent" && (!rules.dismissStaleReviews || rv.commitId === p.head_sha)) result.approvals++; | |
| 718 | + // Only people's approvals count toward requirements: agents' reviews are advisory, and a | |
| 719 | + // workflow's token (kind "system", or "agent" for code an agent wrote) never approves its own way in. | |
| 720 | + if (rv.state === "APPROVED" && rv.author.kind === "user" && (!rules.dismissStaleReviews || rv.commitId === p.head_sha)) result.approvals++; | |
| 709 | 721 | if (rv.state === "CHANGES_REQUESTED") result.changesRequested.push(rv.author.login); |
| 710 | 722 | } |
| 711 | 723 | // Commit statuses count as checks too: a required check matches a status context. |
| @@ -728,6 +740,8 @@ | ||
| 728 | 740 | if (rules.requireLinearHistory && mb !== baseSha) result.blockers.push("branch is behind its base; update it (linear history required)"); |
| 729 | 741 | if (caller) { |
| 730 | 742 | if (!atLeast(caller.role, "write")) result.blockers.push("write access is required to merge"); |
| 743 | + const workflowEdit = await this.repo.policy.refuseWorkflowEdit(caller, mb ?? ZERO_OID, p.head_sha); | |
| 744 | + if (workflowEdit) result.blockers.push(workflowEdit); | |
| 731 | 745 | const isDefault = p.base_ref === this.repo.meta.defaultBranch; |
| 732 | 746 | if (caller.kind === "agent" && (isDefault || Object.keys(rules).length) && !rules.agentsMayMerge) result.blockers.push("agents may not merge into this branch; a human must merge"); |
| 733 | 747 | } |
| @@ -744,8 +758,10 @@ | ||
| 744 | 758 | const rules = this.repo.protectionFor(p.base_ref) ?? {}; |
| 745 | 759 | // Admins (people, not agents) may bypass unmet rules, but only when they ask to. |
| 746 | 760 | const mayBypass = caller.role === "admin" && !rules.enforceAdmins && caller.kind === "user" && m.conflicts.length === 0 && !p.draft; |
| 747 | - const adminBypass = mayBypass && !!(input.bypassRules || input.bypassMergeQueue); | |
| 748 | - if (rules.mergeQueue && !(adminBypass && input.bypassMergeQueue)) { | |
| 761 | + // Each bypass covers only what it names: bypass_merge_queue skips the queue, bypass_rules skips unmet rules. | |
| 762 | + const bypassQueue = mayBypass && !!input.bypassMergeQueue; | |
| 763 | + const adminBypass = mayBypass && !!input.bypassRules; | |
| 764 | + if (rules.mergeQueue && !bypassQueue) { | |
| 749 | 765 | throw new HttpError(405, `${p.base_ref} requires a merge queue: add the pull request to the queue${mayBypass ? " (or, as an admin, bypass the queue with bypass_merge_queue)" : ""}`, { blockers: [`${p.base_ref} merges through a merge queue`] }); |
| 750 | 766 | } |
| 751 | 767 | if (!m.mergeable && !adminBypass) throw new HttpError(405, `pull request is not mergeable: ${m.blockers.join("; ")}${mayBypass ? " (as an admin you can bypass with bypass_rules)" : ""}`, { blockers: m.blockers }); |
| @@ -754,9 +770,12 @@ | ||
| 754 | 770 | const store = this.repo.store; |
| 755 | 771 | const method = input.method ?? (rules.requireLinearHistory ? "squash" : "merge"); |
| 756 | 772 | if (method === "merge" && rules.requireLinearHistory && !adminBypass) throw new HttpError(405, `${p.base_ref} requires linear history; use squash or rebase`); |
| 757 | - if (adminBypass && !m.mergeable) { | |
| 773 | + const linearBypassed = method === "merge" && !!rules.requireLinearHistory; | |
| 774 | + const queueBypassed = bypassQueue && !!rules.mergeQueue; | |
| 775 | + if ((adminBypass && (!m.mergeable || linearBypassed)) || queueBypassed) { | |
| 758 | 776 | const meta = this.repo.meta; |
| 759 | - this.repo.emit({ type: "audit", action: "pull_request.merge_bypassed", actor: caller, target: `${meta.owner}/${meta.name}#${number}`, detail: { blockers: m.blockers, base: p.base_ref }, at: now() }); | |
| 777 | + const blockers = [...(adminBypass ? m.blockers : []), ...(linearBypassed ? ["linear history required"] : []), ...(queueBypassed ? ["merge queue"] : [])]; | |
| 778 | + this.repo.emit({ type: "audit", action: "pull_request.merge_bypassed", actor: caller, target: `${meta.owner}/${meta.name}#${number}`, detail: { blockers, base: p.base_ref }, at: now() }); | |
| 760 | 779 | } |
| 761 | 780 | let newSha: string; |
| 762 | 781 | const ts = Math.floor(Date.now() / 1000); |
modifiedapps/diffinery/src/repo/deployments.ts+19−14View file
| @@ -175,8 +175,10 @@ | ||
| 175 | 175 | if (!ref) throw new HttpError(422, "ref is required", { errors: [{ resource: "Deployment", code: "missing_field", field: "ref" }] }); |
| 176 | 176 | const sha = await this.repo.resolve(ref); |
| 177 | 177 | if (!sha) throw new HttpError(422, `no ref found for: ${ref}`, { errors: [{ resource: "Deployment", code: "invalid", field: "ref" }] }); |
| 178 | - const environment = (input.environment ?? "production").trim(); | |
| 179 | - if (!ENV_NAME.test(environment)) throw new HttpError(422, "invalid environment name"); | |
| 178 | + const rawEnvironment = (input.environment ?? "production").trim(); | |
| 179 | + if (!ENV_NAME.test(rawEnvironment)) throw new HttpError(422, "invalid environment name"); | |
| 180 | + // Environment names are case-insensitive, as on GitHub: reuse the existing spelling. | |
| 181 | + const environment = this.envRow(rawEnvironment)?.name ?? rawEnvironment; | |
| 180 | 182 | // Like GitHub: without required_contexts every context on the commit must have passed; [] skips the check. |
| 181 | 183 | // Deployments a workflow job makes for its `environment:` are not checked (the job's own run is still going). |
| 182 | 184 | if (jobId === null && (input.required_contexts === undefined || input.required_contexts.length)) { |
| @@ -198,7 +200,7 @@ | ||
| 198 | 200 | input.description ? String(input.description).slice(0, 1000) : null, |
| 199 | 201 | typeof input.payload === "string" ? input.payload : JSON.stringify(input.payload ?? {}), |
| 200 | 202 | input.transient_environment ? 1 : 0, |
| 201 | - (input.production_environment ?? environment === "production") ? 1 : 0, | |
| 203 | + (input.production_environment ?? environment.toLowerCase() === "production") ? 1 : 0, | |
| 202 | 204 | caller.id, |
| 203 | 205 | caller.login, |
| 204 | 206 | caller.kind, |
| @@ -216,7 +218,7 @@ | ||
| 216 | 218 | const args: unknown[] = []; |
| 217 | 219 | for (const [col, v] of [["sha", filter.sha], ["ref", filter.ref], ["task", filter.task], ["environment", filter.environment]] as const) { |
| 218 | 220 | if (v) { |
| 219 | - where.push(`${col} = ?`); | |
| 221 | + where.push(col === "environment" ? "environment = ? COLLATE NOCASE" : `${col} = ?`); | |
| 220 | 222 | args.push(v); |
| 221 | 223 | } |
| 222 | 224 | } |
| @@ -233,7 +235,7 @@ | ||
| 233 | 235 | remove(id: number, caller: Caller): void { |
| 234 | 236 | if (!atLeast(caller.role, "write")) throw new HttpError(403, "write access is required to delete deployments"); |
| 235 | 237 | const d = this.get(id); |
| 236 | - const others = this.sql.exec<{ n: number }>("SELECT count(*) AS n FROM deployments WHERE environment = ? AND id != ?", d.environment, id).one().n; | |
| 238 | + const others = this.sql.exec<{ n: number }>("SELECT count(*) AS n FROM deployments WHERE environment = ? COLLATE NOCASE AND id != ?", d.environment, id).one().n; | |
| 237 | 239 | if (d.state === "success" && others > 0) throw new HttpError(422, "We cannot delete an active deployment unless it is the only deployment in a given environment."); |
| 238 | 240 | this.sql.exec("DELETE FROM deployment_statuses WHERE deployment_id = ?", id); |
| 239 | 241 | this.sql.exec("DELETE FROM deployments WHERE id = ?", id); |
| @@ -250,10 +252,11 @@ | ||
| 250 | 252 | if (!STATES.has(state)) throw new HttpError(422, "state must be one of error, failure, inactive, in_progress, queued, pending, success", { errors: [{ resource: "DeploymentStatus", code: "invalid", field: "state" }] }); |
| 251 | 253 | const urls = [input.log_url ?? input.target_url ?? "", input.environment_url ?? ""]; |
| 252 | 254 | for (const u of urls) if (u && !/^https?:\/\//i.test(u)) throw new HttpError(422, "URLs must be http(s)"); |
| 253 | - const environment = input.environment?.trim() || d.environment; | |
| 254 | - if (environment !== d.environment) { | |
| 255 | + let environment = input.environment?.trim() || d.environment; | |
| 256 | + if (environment.toLowerCase() === d.environment.toLowerCase()) environment = d.environment; | |
| 257 | + else { | |
| 255 | 258 | if (!ENV_NAME.test(environment)) throw new HttpError(422, "invalid environment name"); |
| 256 | - this.ensureEnvironment(environment); | |
| 259 | + environment = this.ensureEnvironment(environment).name; | |
| 257 | 260 | this.sql.exec("UPDATE deployments SET environment = ? WHERE id = ?", environment, id); |
| 258 | 261 | } |
| 259 | 262 | const t = now(); |
| @@ -273,8 +276,8 @@ | ||
| 273 | 276 | const statusId = this.sql.exec<{ id: number }>("SELECT last_insert_rowid() AS id").one().id; |
| 274 | 277 | this.sql.exec("UPDATE deployments SET updated_at = ? WHERE id = ?", t, id); |
| 275 | 278 | if (state === "success" && input.auto_inactive !== false) { |
| 276 | - // Older non-transient, non-production deployments to the same environment become inactive. | |
| 277 | - const older = this.sql.exec<{ id: number }>("SELECT id FROM deployments WHERE environment = ? AND id != ? AND transient = 0 AND production = 0", environment, id).toArray(); | |
| 279 | + // Older non-transient, non-production deployments to the same environment become inactive (newer ones stay as they are). | |
| 280 | + const older = this.sql.exec<{ id: number }>("SELECT id FROM deployments WHERE environment = ? COLLATE NOCASE AND id < ? AND transient = 0 AND production = 0", environment, id).toArray(); | |
| 278 | 281 | for (const o of older) { |
| 279 | 282 | const last = this.sql.exec<{ state: string }>("SELECT state FROM deployment_statuses WHERE deployment_id = ? ORDER BY id DESC LIMIT 1", o.id).toArray()[0]; |
| 280 | 283 | if (last?.state === "success") { |
| @@ -468,9 +471,9 @@ | ||
| 468 | 471 | * returns the environment's secrets and variables for the job. |
| 469 | 472 | */ |
| 470 | 473 | async jobStarted(job: { id: string; ref: string; sha: string; logUrl: string; actor: Person }, environment: { name: string; url?: string }): Promise<{ secrets: Record<string, string>; vars: Record<string, string> }> { |
| 471 | - const name = environment.name.trim(); | |
| 472 | - if (!ENV_NAME.test(name)) throw new HttpError(422, `invalid environment name '${name}'`); | |
| 473 | - this.ensureEnvironment(name); | |
| 474 | + const raw = environment.name.trim(); | |
| 475 | + if (!ENV_NAME.test(raw)) throw new HttpError(422, `invalid environment name '${raw}'`); | |
| 476 | + const name = this.ensureEnvironment(raw).name; | |
| 474 | 477 | if (!this.refAllowed(name, job.ref)) throw new HttpError(403, `${job.ref.replace(/^refs\/(heads|tags)\//, "")} is not allowed to deploy to ${name} due to environment protection rules.`); |
| 475 | 478 | const system: Caller = { ...job.actor, kind: job.actor.kind as Caller["kind"], role: "write" }; |
| 476 | 479 | if (!this.sql.exec("SELECT 1 FROM deployments WHERE job_id = ?", job.id).toArray().length) { |
| @@ -481,7 +484,7 @@ | ||
| 481 | 484 | job.ref.replace(/^refs\/(heads|tags)\//, ""), |
| 482 | 485 | name, |
| 483 | 486 | name, |
| 484 | - name === "production" ? 1 : 0, | |
| 487 | + name.toLowerCase() === "production" ? 1 : 0, | |
| 485 | 488 | job.actor.id, |
| 486 | 489 | job.actor.login, |
| 487 | 490 | job.actor.kind, |
| @@ -490,6 +493,8 @@ | ||
| 490 | 493 | t, |
| 491 | 494 | ); |
| 492 | 495 | const id = this.sql.exec<{ id: number }>("SELECT last_insert_rowid() AS id").one().id; |
| 496 | + const ref = job.ref.replace(/^refs\/(heads|tags)\//, ""); | |
| 497 | + this.emit("deployment", "created", { deployment: { id, sha: job.sha, ref, task: "deploy", environment: name, description: null, payload: {}, created_at: t } }, system); | |
| 493 | 498 | this.createStatus(id, { state: "in_progress", log_url: job.logUrl, environment_url: environment.url && /^https?:\/\//i.test(environment.url) ? environment.url : undefined }, system); |
| 494 | 499 | } |
| 495 | 500 | const secrets: Record<string, string> = {}; |
addedapps/diffinery/src/repo/event-payloads.ts+226−0View file
| @@ -1,0 +1,226 @@ | ||
| 1 | +import type { Comment, Issue } from "./collab.js"; | |
| 2 | +import type { RepoMeta } from "./types.js"; | |
| 3 | + | |
| 4 | +/** | |
| 5 | + * Webhook-style event payloads (what a job reads from GITHUB_EVENT_PATH and | |
| 6 | + * `github.event`), shaped like GitHub's so actions and scripts that read | |
| 7 | + * `context.payload` work unmodified: `repository`, `sender` and `organization` | |
| 8 | + * objects with their URLs, full `pull_request` objects (with `head.repo` and | |
| 9 | + * `base.repo`), and `issue`, `comment` and `release` objects. Triggers fill in | |
| 10 | + * what they know; `completePayload` adds the rest when the run is created. | |
| 11 | + */ | |
| 12 | + | |
| 13 | +export type PayloadSite = { publicUrl: string; meta: RepoMeta; visibility?: "public" | "private" }; | |
| 14 | +type Who = { id?: string; login: string; kind?: string }; | |
| 15 | + | |
| 16 | +const api = (s: PayloadSite) => `${s.publicUrl}/api/v3`; | |
| 17 | + | |
| 18 | +/** A user object (`sender`, `user`, `owner`). Agents and the system appear as bots. */ | |
| 19 | +export function userPayload(s: PayloadSite, who: Who): Record<string, unknown> { | |
| 20 | + const bot = who.kind === "agent" || who.kind === "system" || who.login.endsWith("[bot]"); | |
| 21 | + return { | |
| 22 | + login: who.login, | |
| 23 | + id: who.id ?? who.login, | |
| 24 | + node_id: who.id ?? who.login, | |
| 25 | + type: bot ? "Bot" : "User", | |
| 26 | + site_admin: false, | |
| 27 | + avatar_url: `${s.publicUrl}/${encodeURIComponent(who.login)}.png`, | |
| 28 | + html_url: `${s.publicUrl}/${encodeURIComponent(who.login)}`, | |
| 29 | + url: `${api(s)}/users/${encodeURIComponent(who.login)}`, | |
| 30 | + }; | |
| 31 | +} | |
| 32 | + | |
| 33 | +export function organizationPayload(s: PayloadSite): Record<string, unknown> | undefined { | |
| 34 | + const m = s.meta; | |
| 35 | + if (m.ownerKind !== "org") return undefined; | |
| 36 | + return { login: m.owner, id: m.ownerId, node_id: m.ownerId, url: `${api(s)}/orgs/${m.owner}`, repos_url: `${api(s)}/orgs/${m.owner}/repos`, avatar_url: `${s.publicUrl}/${m.owner}.png`, description: null }; | |
| 37 | +} | |
| 38 | + | |
| 39 | +export function repositoryPayload(s: PayloadSite): Record<string, unknown> { | |
| 40 | + const m = s.meta; | |
| 41 | + const full = `${m.owner}/${m.name}`; | |
| 42 | + const url = `${api(s)}/repos/${full}`; | |
| 43 | + const html = `${s.publicUrl}/${full}`; | |
| 44 | + const host = new URL(s.publicUrl).host; | |
| 45 | + return { | |
| 46 | + id: m.repoId, | |
| 47 | + node_id: m.repoId, | |
| 48 | + name: m.name, | |
| 49 | + full_name: full, | |
| 50 | + private: s.visibility !== "public", | |
| 51 | + visibility: s.visibility ?? "private", | |
| 52 | + owner: { ...userPayload(s, { id: m.ownerId, login: m.owner }), type: m.ownerKind === "org" ? "Organization" : "User", name: m.owner }, | |
| 53 | + html_url: html, | |
| 54 | + description: null, | |
| 55 | + fork: false, | |
| 56 | + url, | |
| 57 | + archive_url: `${url}/{archive_format}{/ref}`, | |
| 58 | + branches_url: `${url}/branches{/branch}`, | |
| 59 | + commits_url: `${url}/commits{/sha}`, | |
| 60 | + contents_url: `${url}/contents/{+path}`, | |
| 61 | + issues_url: `${url}/issues{/number}`, | |
| 62 | + pulls_url: `${url}/pulls{/number}`, | |
| 63 | + labels_url: `${url}/labels{/name}`, | |
| 64 | + releases_url: `${url}/releases{/id}`, | |
| 65 | + statuses_url: `${url}/statuses/{sha}`, | |
| 66 | + git_url: `git://${host}/${full}.git`, | |
| 67 | + ssh_url: `git@${host}:${full}.git`, | |
| 68 | + clone_url: `${html}.git`, | |
| 69 | + svn_url: html, | |
| 70 | + homepage: null, | |
| 71 | + default_branch: m.defaultBranch, | |
| 72 | + master_branch: m.defaultBranch, | |
| 73 | + archived: false, | |
| 74 | + disabled: false, | |
| 75 | + is_template: false, | |
| 76 | + has_issues: true, | |
| 77 | + has_projects: false, | |
| 78 | + has_wiki: false, | |
| 79 | + has_pages: false, | |
| 80 | + has_downloads: true, | |
| 81 | + allow_forking: false, | |
| 82 | + topics: [], | |
| 83 | + license: null, | |
| 84 | + }; | |
| 85 | +} | |
| 86 | + | |
| 87 | +/** `issue` of the issues and issue_comment events (a pull request carries `pull_request` links, like GitHub). */ | |
| 88 | +export function issuePayload(s: PayloadSite, i: Issue): Record<string, unknown> { | |
| 89 | + const full = `${s.meta.owner}/${s.meta.name}`; | |
| 90 | + const url = `${api(s)}/repos/${full}/issues/${i.number}`; | |
| 91 | + return { | |
| 92 | + id: i.number, | |
| 93 | + node_id: `${s.meta.repoId}:${i.number}`, | |
| 94 | + number: i.number, | |
| 95 | + title: i.title, | |
| 96 | + body: i.body, | |
| 97 | + state: i.state, | |
| 98 | + state_reason: i.stateReason, | |
| 99 | + locked: false, | |
| 100 | + user: userPayload(s, i.author), | |
| 101 | + labels: i.labels.map((name) => ({ name, url: `${api(s)}/repos/${full}/labels/${encodeURIComponent(name)}` })), | |
| 102 | + assignee: i.assignees[0] ? userPayload(s, { login: i.assignees[0] }) : null, | |
| 103 | + assignees: i.assignees.map((login) => userPayload(s, { login })), | |
| 104 | + comments: i.comments, | |
| 105 | + created_at: i.createdAt, | |
| 106 | + updated_at: i.updatedAt, | |
| 107 | + closed_at: i.closedAt, | |
| 108 | + author_association: "MEMBER", | |
| 109 | + url, | |
| 110 | + repository_url: `${api(s)}/repos/${full}`, | |
| 111 | + comments_url: `${url}/comments`, | |
| 112 | + html_url: `${s.publicUrl}/${full}/${i.kind === "pull" ? "pull" : "issues"}/${i.number}`, | |
| 113 | + ...(i.kind === "pull" ? { pull_request: { url: `${api(s)}/repos/${full}/pulls/${i.number}`, html_url: `${s.publicUrl}/${full}/pull/${i.number}`, merged_at: i.pull?.mergedAt ?? null } } : {}), | |
| 114 | + }; | |
| 115 | +} | |
| 116 | + | |
| 117 | +export function commentPayload(s: PayloadSite, issueNumber: number, c: Comment): Record<string, unknown> { | |
| 118 | + const full = `${s.meta.owner}/${s.meta.name}`; | |
| 119 | + return { | |
| 120 | + id: c.id, | |
| 121 | + node_id: c.id, | |
| 122 | + body: c.body, | |
| 123 | + user: userPayload(s, c.author), | |
| 124 | + created_at: c.createdAt, | |
| 125 | + updated_at: c.updatedAt, | |
| 126 | + author_association: "MEMBER", | |
| 127 | + url: `${api(s)}/repos/${full}/issues/comments/${c.id}`, | |
| 128 | + html_url: `${s.publicUrl}/${full}/issues/${issueNumber}#issuecomment-${c.id}`, | |
| 129 | + issue_url: `${api(s)}/repos/${full}/issues/${issueNumber}`, | |
| 130 | + }; | |
| 131 | +} | |
| 132 | + | |
| 133 | +export type ReleaseFacts = { | |
| 134 | + id: string | number; | |
| 135 | + tagName: string; | |
| 136 | + targetCommitish: string; | |
| 137 | + name: string | null; | |
| 138 | + body: string | null; | |
| 139 | + draft: boolean; | |
| 140 | + prerelease: boolean; | |
| 141 | + createdAt: string; | |
| 142 | + publishedAt: string | null; | |
| 143 | + author: Who; | |
| 144 | +}; | |
| 145 | + | |
| 146 | +export function releasePayload(s: PayloadSite, r: ReleaseFacts): Record<string, unknown> { | |
| 147 | + const full = `${s.meta.owner}/${s.meta.name}`; | |
| 148 | + const url = `${api(s)}/repos/${full}/releases/${r.id}`; | |
| 149 | + return { | |
| 150 | + id: r.id, | |
| 151 | + node_id: String(r.id), | |
| 152 | + tag_name: r.tagName, | |
| 153 | + target_commitish: r.targetCommitish, | |
| 154 | + name: r.name, | |
| 155 | + body: r.body, | |
| 156 | + draft: r.draft, | |
| 157 | + prerelease: r.prerelease, | |
| 158 | + created_at: r.createdAt, | |
| 159 | + published_at: r.publishedAt, | |
| 160 | + author: userPayload(s, r.author), | |
| 161 | + assets: [], | |
| 162 | + url, | |
| 163 | + assets_url: `${url}/assets`, | |
| 164 | + upload_url: `${url}/assets{?name,label}`, | |
| 165 | + html_url: `${s.publicUrl}/${full}/releases/tag/${encodeURIComponent(r.tagName)}`, | |
| 166 | + tarball_url: `${api(s)}/repos/${full}/tarball/${encodeURIComponent(r.tagName)}`, | |
| 167 | + zipball_url: `${api(s)}/repos/${full}/zipball/${encodeURIComponent(r.tagName)}`, | |
| 168 | + }; | |
| 169 | +} | |
| 170 | + | |
| 171 | +const isObj = (v: unknown): v is Record<string, unknown> => !!v && typeof v === "object" && !Array.isArray(v); | |
| 172 | + | |
| 173 | +/** `pull_request` filled out to GitHub's shape around what the trigger set. */ | |
| 174 | +function completePull(s: PayloadSite, pr: Record<string, unknown>, sender: Who): Record<string, unknown> { | |
| 175 | + const full = `${s.meta.owner}/${s.meta.name}`; | |
| 176 | + const n = pr.number as number; | |
| 177 | + const url = `${api(s)}/repos/${full}/pulls/${n}`; | |
| 178 | + const side = (v: unknown) => { | |
| 179 | + const b = isObj(v) ? v : {}; | |
| 180 | + const ref = typeof b.ref === "string" ? b.ref : ""; | |
| 181 | + return { label: `${s.meta.owner}:${ref}`, ref, sha: b.sha ?? null, user: userPayload(s, { id: s.meta.ownerId, login: s.meta.owner }), ...b, repo: repositoryPayload(s) }; | |
| 182 | + }; | |
| 183 | + const user = isObj(pr.user) && typeof pr.user.login === "string" ? userPayload(s, { login: pr.user.login }) : userPayload(s, sender); | |
| 184 | + return { | |
| 185 | + id: n, | |
| 186 | + node_id: `${s.meta.repoId}:${n}`, | |
| 187 | + state: "open", | |
| 188 | + locked: false, | |
| 189 | + merged: false, | |
| 190 | + draft: false, | |
| 191 | + labels: [], | |
| 192 | + assignees: [], | |
| 193 | + requested_reviewers: [], | |
| 194 | + author_association: "MEMBER", | |
| 195 | + url, | |
| 196 | + html_url: `${s.publicUrl}/${full}/pull/${n}`, | |
| 197 | + diff_url: `${s.publicUrl}/${full}/pull/${n}.diff`, | |
| 198 | + patch_url: `${s.publicUrl}/${full}/pull/${n}.patch`, | |
| 199 | + issue_url: `${api(s)}/repos/${full}/issues/${n}`, | |
| 200 | + commits_url: `${url}/commits`, | |
| 201 | + review_comments_url: `${url}/comments`, | |
| 202 | + comments_url: `${api(s)}/repos/${full}/issues/${n}/comments`, | |
| 203 | + ...pr, | |
| 204 | + user, | |
| 205 | + head: side(pr.head), | |
| 206 | + base: side(pr.base), | |
| 207 | + }; | |
| 208 | +} | |
| 209 | + | |
| 210 | +/** | |
| 211 | + * Adds GitHub's common fields to a trigger's payload: full `repository`, | |
| 212 | + * `sender` and (for organisation-owned repositories) `organization`, plus a | |
| 213 | + * complete `pull_request`. Fields the trigger set are kept. | |
| 214 | + */ | |
| 215 | +export function completePayload(s: PayloadSite, payload: Record<string, unknown>, sender: Who): Record<string, unknown> { | |
| 216 | + const out: Record<string, unknown> = { ...payload }; | |
| 217 | + // A trigger's own repository object (the directory's, with its real visibility) wins over the generic one. | |
| 218 | + out.repository = { ...repositoryPayload(s), ...(isObj(payload.repository) ? payload.repository : {}) }; | |
| 219 | + const senderLogin = isObj(payload.sender) && typeof payload.sender.login === "string" ? payload.sender.login : sender.login; | |
| 220 | + out.sender = userPayload(s, senderLogin === sender.login ? sender : { login: senderLogin }); | |
| 221 | + const org = organizationPayload(s); | |
| 222 | + if (org) out.organization = org; | |
| 223 | + if (isObj(payload.pull_request)) out.pull_request = completePull(s, payload.pull_request, sender); | |
| 224 | + if (isObj(payload.pusher) && typeof payload.pusher.name === "string") out.pusher = { email: null, ...payload.pusher }; | |
| 225 | + return out; | |
| 226 | +} |
modifiedapps/diffinery/src/repo/extra-triggers.ts+4−1View file
| @@ -16,10 +16,13 @@ | ||
| 16 | 16 | ): Promise<number> { |
| 17 | 17 | if (caller.kind === "system" && caller.id.startsWith("job:") && ev.name !== "repository_dispatch") return 0; |
| 18 | 18 | let started = 0; |
| 19 | + let payload: Record<string, unknown> | undefined; | |
| 19 | 20 | try { |
| 20 | 21 | for (const w of await repo.actions.loadWorkflows(ev.sha)) { |
| 21 | 22 | if (!w.workflow || !matchesTrigger(w.workflow, { name: ev.name, action: ev.action })) continue; |
| 22 | - await repo.actions.createRun(w.workflow, { event: ev.name, ref: ev.ref, sha: ev.sha, payload: ev.payload, caller, title: ev.title }); | |
| 23 | + // Every event payload carries GitHub's repository object (with its real visibility) and a sender. | |
| 24 | + payload ??= { ...ev.payload, repository: await repo.actions.repositoryPayload(), sender: ev.payload.sender ?? { login: caller.login, type: caller.kind === "user" ? "User" : "Bot" } }; | |
| 25 | + await repo.actions.createRun(w.workflow, { event: ev.name, ref: ev.ref, sha: ev.sha, payload, caller, title: ev.title }); | |
| 23 | 26 | started++; |
| 24 | 27 | } |
| 25 | 28 | } catch (e) { |
modifiedapps/diffinery/src/repo/gitdata.ts+9−7View file
| @@ -25,6 +25,7 @@ | ||
| 25 | 25 | import { R2Sink } from "../lib/r2-sink.js"; |
| 26 | 26 | import { unsafeTreeName, unsafeTreePath } from "../lib/tree-names.js"; |
| 27 | 27 | import { HttpError, now } from "../lib/util.js"; |
| 28 | +import { archiveKey } from "../lib/storage-keys.js"; | |
| 28 | 29 | import type { Check } from "./collab.js"; |
| 29 | 30 | import { toInfo, type CommitInfo, type RepoDO } from "./RepoDO.js"; |
| 30 | 31 | import { atLeast, type Caller } from "./types.js"; |
| @@ -105,11 +106,11 @@ | ||
| 105 | 106 | } |
| 106 | 107 | |
| 107 | 108 | /** Same push protection as git pushes: blobs with credentials are refused. */ |
| 108 | - private scan(oid: string, content: Uint8Array): void { | |
| 109 | + private scan(oid: string, content: Uint8Array, caller: Caller): void { | |
| 109 | 110 | const hits = scanForSecrets([{ oid, content }]); |
| 110 | 111 | if (hits.length) { |
| 111 | 112 | const m = this.repo.meta; |
| 112 | - this.repo.emit({ type: "audit", action: "api.blocked_secret", actor: { id: "api", login: "api", kind: "system" }, target: `${m.owner}/${m.name}`, detail: { kinds: hits.map((h) => h.kind) }, at: now() }); | |
| 113 | + this.repo.emit({ type: "audit", action: "api.blocked_secret", actor: caller, target: `${m.owner}/${m.name}`, detail: { kinds: hits.map((h) => h.kind), blob: oid }, at: now() }); | |
| 113 | 114 | throw new HttpError(422, `push protection: possible secret detected (${hits.map((h) => h.kind).join(", ")})`); |
| 114 | 115 | } |
| 115 | 116 | } |
| @@ -129,7 +130,7 @@ | ||
| 129 | 130 | async createBlob(content: Uint8Array, caller: Caller): Promise<{ sha: string; size: number }> { |
| 130 | 131 | this.requireWrite(caller); |
| 131 | 132 | const oid = await hashObject("blob", content); |
| 132 | - this.scan(oid, content); | |
| 133 | + this.scan(oid, content, caller); | |
| 133 | 134 | await this.store.writeKnown(oid, "blob", content); |
| 134 | 135 | return { sha: oid, size: content.length }; |
| 135 | 136 | } |
| @@ -300,7 +301,6 @@ | ||
| 300 | 301 | if (existing && input.sha !== existing.oid) throw new HttpError(409, `${input.path} does not match ${input.sha}`); |
| 301 | 302 | if (!existing && input.sha) throw new HttpError(409, `${input.path} does not exist at ${branch}`); |
| 302 | 303 | const oid = await hashObject("blob", input.content); |
| 303 | - this.scan(oid, input.content); | |
| 304 | 304 | const res = await this.repo.commitFiles({ branch, message: input.message, changes: [{ path: input.path, content: input.content }], expectedHeadSha: head ?? undefined, author: input.author, committer: input.committer }, caller); |
| 305 | 305 | return { created: !existing, commit: await this.repo.commitInfo(res.sha), blob: { sha: oid, size: input.content.length } }; |
| 306 | 306 | } |
| @@ -388,7 +388,8 @@ | ||
| 388 | 388 | } |
| 389 | 389 | |
| 390 | 390 | archiveKey(sha: string, format: "zip" | "tar.gz"): string { |
| 391 | - return `repos/${this.repo.meta.repoId}/archives/${sha}.${format}`; | |
| 391 | + // Outside repos/: a cache the lifecycle rules may expire (lib/storage-keys.ts). | |
| 392 | + return archiveKey(this.repo.meta.repoId, sha, format); | |
| 392 | 393 | } |
| 393 | 394 | |
| 394 | 395 | /** |
| @@ -528,11 +529,12 @@ | ||
| 528 | 529 | |
| 529 | 530 | // ---------------------------------------------------------------- check runs and suites |
| 530 | 531 | |
| 531 | - /** A check run by GitHub's numeric id or Diffinery's own. */ | |
| 532 | + /** A check run by GitHub's numeric id, Diffinery's own, or the Actions job that reports it (a job's `check_run_url`). */ | |
| 532 | 533 | check(raw: string): Check { |
| 533 | 534 | const row = /^\d{1,15}$/.test(raw) |
| 534 | 535 | ? this.sql.exec<{ sha: string; id: string }>("SELECT sha, id FROM checks WHERE seq = ?", Number(raw)).toArray()[0] |
| 535 | - : this.sql.exec<{ sha: string; id: string }>("SELECT sha, id FROM checks WHERE id = ?", raw).toArray()[0]; | |
| 536 | + : (this.sql.exec<{ sha: string; id: string }>("SELECT sha, id FROM checks WHERE id = ?", raw).toArray()[0] ?? | |
| 537 | + this.sql.exec<{ sha: string; id: string }>("SELECT sha, id FROM checks WHERE job_id = ? ORDER BY seq DESC LIMIT 1", raw).toArray()[0]); | |
| 536 | 538 | const c = row ? this.repo.collab.checks(row.sha).find((x) => x.id === row.id) : undefined; |
| 537 | 539 | if (!c) throw new HttpError(404, "check run not found"); |
| 538 | 540 | return c; |
addedapps/diffinery/src/repo/job-graph.ts+104−0View file
| @@ -1,0 +1,104 @@ | ||
| 1 | +import type { Conclusion } from "@diffinery/shared"; | |
| 2 | +import { evaluateValue, type ExprContext, type StatusFns } from "@diffinery/workflow-yaml"; | |
| 3 | + | |
| 4 | +/** | |
| 5 | + * Pure pieces of job scheduling, GitHub-compatible: the dependency graph a run | |
| 6 | + * walks, the status functions a job's `if:` sees, how matrix legs combine into | |
| 7 | + * one `needs.<job>.result`, and `runs-on` evaluated for one matrix leg. | |
| 8 | + */ | |
| 9 | + | |
| 10 | +/** A job and the jobs it waits for. */ | |
| 11 | +export type JobNode = { key: string; needs: string[] }; | |
| 12 | + | |
| 13 | +/** | |
| 14 | + * Every job (or reusable workflow call) that `keys` depend on, directly or | |
| 15 | + * through their own `needs`: GitHub's "ancestor" jobs. `needs` maps a key to | |
| 16 | + * the keys it declares. | |
| 17 | + */ | |
| 18 | +export function ancestors(needs: ReadonlyMap<string, readonly string[]>, keys: readonly string[]): string[] { | |
| 19 | + const seen = new Set<string>(); | |
| 20 | + const visit = (k: string) => { | |
| 21 | + if (seen.has(k)) return; | |
| 22 | + seen.add(k); | |
| 23 | + for (const n of needs.get(k) ?? []) visit(n); | |
| 24 | + }; | |
| 25 | + for (const k of keys) visit(k); | |
| 26 | + return [...seen]; | |
| 27 | +} | |
| 28 | + | |
| 29 | +/** | |
| 30 | + * Status functions for a job's `if:`, from the results of all of its ancestors. | |
| 31 | + * Like GitHub: success() needs every ancestor to have succeeded (a skipped one | |
| 32 | + * counts against it, so a skip cascades down the chain), failure() is true when | |
| 33 | + * any ancestor failed, and cancelled() when the run was cancelled. | |
| 34 | + */ | |
| 35 | +export function jobStatus(ancestorResults: readonly Conclusion[], cancelled: boolean): StatusFns { | |
| 36 | + return { | |
| 37 | + success: () => !cancelled && ancestorResults.every((r) => r === "success"), | |
| 38 | + failure: () => ancestorResults.some((r) => r === "failure"), | |
| 39 | + cancelled: () => cancelled, | |
| 40 | + always: () => true, | |
| 41 | + }; | |
| 42 | +} | |
| 43 | + | |
| 44 | +/** | |
| 45 | + * The result a job reports to the jobs that need it, from its matrix legs: any | |
| 46 | + * failure fails it, else any cancellation cancels it; a job that never started, | |
| 47 | + * or whose legs were all skipped, is skipped. A leg with `continue-on-error` | |
| 48 | + * that failed counts as a success, as on GitHub. | |
| 49 | + */ | |
| 50 | +export function combineResults(legs: ReadonlyArray<{ conclusion: Conclusion | "timed_out" | null; continueOnError?: boolean }>): Conclusion { | |
| 51 | + if (legs.length === 0 || legs.every((l) => l.conclusion === "skipped")) return "skipped"; | |
| 52 | + let result: Conclusion = "success"; | |
| 53 | + for (const l of legs) { | |
| 54 | + let c: Conclusion = l.conclusion === "timed_out" ? "failure" : (l.conclusion ?? "failure"); | |
| 55 | + if (c === "failure" && l.continueOnError) c = "success"; | |
| 56 | + if (c === "failure") result = "failure"; | |
| 57 | + else if (c === "cancelled" && result !== "failure") result = "cancelled"; | |
| 58 | + } | |
| 59 | + return result; | |
| 60 | +} | |
| 61 | + | |
| 62 | +/** | |
| 63 | + * `runs-on` labels for one matrix leg. Each entry may be an expression (github, | |
| 64 | + * needs, strategy, matrix, vars and inputs are available, as on GitHub) that | |
| 65 | + * yields a label or a list of labels. | |
| 66 | + */ | |
| 67 | +export function runnerLabels(runsOn: readonly string[], ctx: ExprContext): string[] { | |
| 68 | + const out: string[] = []; | |
| 69 | + for (const entry of runsOn) { | |
| 70 | + const v = evaluateValue(entry, ctx); | |
| 71 | + for (const label of Array.isArray(v) ? v : [v]) { | |
| 72 | + if (label === null || label === undefined || typeof label === "object") continue; | |
| 73 | + const s = String(label).trim(); | |
| 74 | + if (s) out.push(s); | |
| 75 | + } | |
| 76 | + } | |
| 77 | + return out; | |
| 78 | +} | |
| 79 | + | |
| 80 | +/** | |
| 81 | + * Walk a job graph: `start` runs for each job as soon as every job it needs has | |
| 82 | + * finished (its own needs only, not a whole "level"), and the returned promise | |
| 83 | + * settles once every job has. Needs outside the graph are ignored; the graph | |
| 84 | + * must have no cycles (the planner rejects them). | |
| 85 | + */ | |
| 86 | +export async function runGraph(nodes: readonly JobNode[], start: (key: string) => Promise<void>): Promise<void> { | |
| 87 | + const byKey = new Map(nodes.map((n) => [n.key, n])); | |
| 88 | + const done = new Map<string, Promise<void>>(); | |
| 89 | + const visit = (key: string): Promise<void> => { | |
| 90 | + let p = done.get(key); | |
| 91 | + if (!p) { | |
| 92 | + const needs = (byKey.get(key)?.needs ?? []).filter((n) => byKey.has(n)); | |
| 93 | + p = Promise.all(needs.map(visit)).then(() => start(key)); | |
| 94 | + done.set(key, p); | |
| 95 | + } | |
| 96 | + return p; | |
| 97 | + }; | |
| 98 | + await Promise.all(nodes.map((n) => visit(n.key))); | |
| 99 | +} | |
| 100 | + | |
| 101 | +/** The graph of a plan made before plans carried one: each level waits for the level before it. */ | |
| 102 | +export function graphFromLevels(levels: readonly string[][]): JobNode[] { | |
| 103 | + return levels.flatMap((level, i) => level.map((key) => ({ key, needs: i > 0 ? [...levels[i - 1]!] : [] }))); | |
| 104 | +} |
addedapps/diffinery/src/repo/runtime.ts+501−0View file
| @@ -1,0 +1,501 @@ | ||
| 1 | +import type { Env } from "../env.js"; | |
| 2 | +import { dropStaged, finalKey, runComposeStep, stagingPrefix, stepKeys, type BlobBucket, type ComposeStep } from "../lib/blob-store.js"; | |
| 3 | +import { READ_ALL, roleForPermissions, type JobPermissions } from "../lib/job-permissions.js"; | |
| 4 | +import { HttpError, now } from "../lib/util.js"; | |
| 5 | +import type { RepoDO } from "./RepoDO.js"; | |
| 6 | +import type { Role } from "./types.js"; | |
| 7 | + | |
| 8 | +/** | |
| 9 | + * Metadata for the Actions runtime services: GitHub's cache service (keys, | |
| 10 | + * versions and branch scopes) and artifact service (per-run artifacts with | |
| 11 | + * numeric ids and retention). The Worker (http/actions-runtime.ts) handles the | |
| 12 | + * wire protocols and moves the bytes; this module decides what may be read or | |
| 13 | + * written. Every call made with a runtime token names the job, and works only | |
| 14 | + * while that job is running. | |
| 15 | + */ | |
| 16 | + | |
| 17 | +export type CacheEntry = { id: number; key: string; version: string; scope: string; size: number; createdAt: string; lastUsed: string; etag: string | null }; | |
| 18 | +export type ArtifactEntry = { id: number; runId: string; runNumber: number; headBranch: string; headSha: string; jobId: string; name: string; size: number; digest: string | null; mimeType: string; createdAt: string; updatedAt: string; expiresAt: string; expired: boolean; etag: string | null }; | |
| 19 | + | |
| 20 | +export type JobScope = { | |
| 21 | + runId: string; | |
| 22 | + jobId: string; | |
| 23 | + /** Cache scopes the job may read: its own ref first (also writable), then the PR base and the default branch. */ | |
| 24 | + scopes: string[]; | |
| 25 | +}; | |
| 26 | + | |
| 27 | +type CacheRow = { id: number; key: string; version: string; scope: string; size: number; status: string; job_id: string | null; created_at: string; last_used: string; staged: number; etag: string | null }; | |
| 28 | +type ArtifactRow = { id: number; run_id: string; job_id: string; name: string; size: number; digest: string | null; mime_type: string; status: string; created_at: string; updated_at: string; expires_at: string; staged: number; etag: string | null }; | |
| 29 | + | |
| 30 | +/** Per-repository cache budget, and how long an unused entry is kept (GitHub's limits). */ | |
| 31 | +export const CACHE_LIMIT_BYTES = 10 * 1024 ** 3; | |
| 32 | +const CACHE_IDLE_MS = 7 * 24 * 3600_000; | |
| 33 | +/** Artifact retention: `retention-days` is clamped to 1..90 days, GitHub's default maximum. */ | |
| 34 | +export const MAX_RETENTION_DAYS = 90; | |
| 35 | +/** How long an expired artifact stays listed (as `expired: true`) after its content is deleted. */ | |
| 36 | +const EXPIRED_LISTED_DAYS = 90; | |
| 37 | +/** Artifact storage per repository unless ARTIFACT_QUOTA_BYTES says otherwise. */ | |
| 38 | +export const DEFAULT_ARTIFACT_QUOTA = 10 * 1024 ** 3; | |
| 39 | +/** Largest block, chunk or single-request upload accepted. */ | |
| 40 | +export const MAX_UPLOAD_PART = 256 * 1024 * 1024; | |
| 41 | + | |
| 42 | +const toCache = (r: CacheRow): CacheEntry => ({ id: r.id, key: r.key, version: r.version, scope: r.scope, size: r.size, createdAt: r.created_at, lastUsed: r.last_used, etag: r.etag }); | |
| 43 | + | |
| 44 | +export class Runtime { | |
| 45 | + constructor( | |
| 46 | + private repo: RepoDO, | |
| 47 | + private env: Env, | |
| 48 | + ) {} | |
| 49 | + | |
| 50 | + private get sql() { | |
| 51 | + return this.repo.sql; | |
| 52 | + } | |
| 53 | + | |
| 54 | + private get bucket(): BlobBucket { | |
| 55 | + return this.env.BUCKET as unknown as BlobBucket; | |
| 56 | + } | |
| 57 | + | |
| 58 | + private get repoId(): string { | |
| 59 | + return this.repo.meta.repoId; | |
| 60 | + } | |
| 61 | + | |
| 62 | + /** Artifact storage this repository may use (live artifacts plus uploads in progress). */ | |
| 63 | + get artifactQuota(): number { | |
| 64 | + const n = Number(this.env.ARTIFACT_QUOTA_BYTES); | |
| 65 | + return Number.isSafeInteger(n) && n > 0 ? n : DEFAULT_ARTIFACT_QUOTA; | |
| 66 | + } | |
| 67 | + | |
| 68 | + // ---------------------------------------------------------------- jobs | |
| 69 | + | |
| 70 | + /** The job a runtime token names, if it is still running. */ | |
| 71 | + job(jobId: string, runId?: string): JobScope { | |
| 72 | + const j = this.sql | |
| 73 | + .exec<{ status: string; run_id: string; ref: string; event_payload: string }>("SELECT j.status, j.run_id, r.ref, r.event_payload FROM jobs j JOIN runs r ON r.id = j.run_id WHERE j.id = ?", jobId) | |
| 74 | + .toArray()[0]; | |
| 75 | + if (!j || j.status !== "in_progress" || (runId && runId !== j.run_id)) throw new HttpError(401, "the job is not running"); | |
| 76 | + // Pull requests also read their base branch's caches, and merge groups their target branch's. | |
| 77 | + let base: string | undefined; | |
| 78 | + try { | |
| 79 | + const p = JSON.parse(j.event_payload) as { pull_request?: { base?: { ref?: string } }; merge_group?: { base_ref?: string } }; | |
| 80 | + base = p.pull_request?.base?.ref ?? p.merge_group?.base_ref?.replace(/^refs\/heads\//, ""); | |
| 81 | + } catch { | |
| 82 | + /* no payload */ | |
| 83 | + } | |
| 84 | + const scopes = [j.ref, ...(base ? [`refs/heads/${base}`] : []), `refs/heads/${this.repo.meta.defaultBranch}`]; | |
| 85 | + return { runId: j.run_id, jobId, scopes: [...new Set(scopes)] }; | |
| 86 | + } | |
| 87 | + | |
| 88 | + /** Records what the job token may do (its effective `permissions:`, set when the job starts). */ | |
| 89 | + grant(jobId: string, perms: JobPermissions): void { | |
| 90 | + this.sql.exec("INSERT OR REPLACE INTO job_grants (job_id, role, permissions) VALUES (?, ?, ?)", jobId, roleForPermissions(perms), JSON.stringify(perms)); | |
| 91 | + } | |
| 92 | + | |
| 93 | + /** What a job token may do; read-only for a job that has not started. */ | |
| 94 | + grantFor(jobId: string): { role: Role; permissions: JobPermissions } { | |
| 95 | + const r = this.sql.exec<{ role: Role; permissions: string }>("SELECT role, permissions FROM job_grants WHERE job_id = ?", jobId).toArray()[0]; | |
| 96 | + return r ? { role: r.role, permissions: { ...READ_ALL, ...(JSON.parse(r.permissions) as Partial<JobPermissions>) } } : { role: "read", permissions: { ...READ_ALL } }; | |
| 97 | + } | |
| 98 | + | |
| 99 | + // ---------------------------------------------------------------- cache | |
| 100 | + | |
| 101 | + /** | |
| 102 | + * GitHub's lookup order: in each scope (own ref, PR base, default branch), | |
| 103 | + * an exact match of the primary key, then the newest entry whose key starts | |
| 104 | + * with it, then the same for each restore key. The version must match. | |
| 105 | + */ | |
| 106 | + cacheLookup(jobId: string, keys: string[], version: string): CacheEntry | null { | |
| 107 | + const { scopes } = this.job(jobId); | |
| 108 | + for (const scope of scopes) { | |
| 109 | + for (const key of keys) { | |
| 110 | + const exact = this.sql.exec<CacheRow>("SELECT * FROM rt_caches WHERE key = ? AND version = ? AND scope = ? AND status = 'ready'", key, version, scope).toArray()[0]; | |
| 111 | + const hit = | |
| 112 | + exact ?? | |
| 113 | + this.sql | |
| 114 | + .exec<CacheRow>("SELECT * FROM rt_caches WHERE substr(key, 1, ?) = ? AND version = ? AND scope = ? AND status = 'ready' ORDER BY created_at DESC, id DESC LIMIT 1", key.length, key, version, scope) | |
| 115 | + .toArray()[0]; | |
| 116 | + if (hit) { | |
| 117 | + this.sql.exec("UPDATE rt_caches SET last_used = ? WHERE id = ?", now(), hit.id); | |
| 118 | + return toCache(hit); | |
| 119 | + } | |
| 120 | + } | |
| 121 | + } | |
| 122 | + return null; | |
| 123 | + } | |
| 124 | + | |
| 125 | + /** Reserves `key` in the job's own scope. Returns null when an entry already exists there. */ | |
| 126 | + async cacheReserve(jobId: string, key: string, version: string): Promise<{ id: number } | null> { | |
| 127 | + const { scopes } = this.job(jobId); | |
| 128 | + if (!key || key.length > 512 || key.includes(",")) throw new HttpError(400, "invalid cache key"); | |
| 129 | + if (!/^[0-9a-f]{1,128}$/i.test(version)) throw new HttpError(400, "invalid cache version"); | |
| 130 | + const scope = scopes[0]!; | |
| 131 | + const existing = this.sql.exec<CacheRow>("SELECT * FROM rt_caches WHERE key = ? AND version = ? AND scope = ?", key, version, scope).toArray()[0]; | |
| 132 | + if (existing) { | |
| 133 | + // A reservation abandoned by a job that has finished can be taken over. | |
| 134 | + const holder = existing.job_id ? this.sql.exec<{ status: string }>("SELECT status FROM jobs WHERE id = ?", existing.job_id).toArray()[0] : undefined; | |
| 135 | + if (existing.status === "ready" || holder?.status === "in_progress") return null; | |
| 136 | + this.sql.exec("DELETE FROM rt_caches WHERE id = ?", existing.id); | |
| 137 | + await this.dropObjects("cache", [existing.id]); | |
| 138 | + } | |
| 139 | + await this.evictCaches(); | |
| 140 | + const t = now(); | |
| 141 | + this.sql.exec("INSERT INTO rt_caches (key, version, scope, size, status, job_id, created_at, last_used) VALUES (?, ?, ?, 0, 'reserved', ?, ?, ?)", key, version, scope, jobId, t, t); | |
| 142 | + return { id: this.sql.exec<{ id: number }>("SELECT last_insert_rowid() AS id").one().id }; | |
| 143 | + } | |
| 144 | + | |
| 145 | + /** The job's own reservation, by id or by key and version. */ | |
| 146 | + cacheReservation(jobId: string, ref: { id?: number; key?: string; version?: string }): CacheEntry | null { | |
| 147 | + const row = | |
| 148 | + ref.id !== undefined | |
| 149 | + ? this.sql.exec<CacheRow>("SELECT * FROM rt_caches WHERE id = ?", ref.id).toArray()[0] | |
| 150 | + : this.sql.exec<CacheRow>("SELECT * FROM rt_caches WHERE key = ? AND version = ? AND job_id = ? AND status = 'reserved' ORDER BY id DESC LIMIT 1", ref.key ?? "", ref.version ?? "", jobId).toArray()[0]; | |
| 151 | + if (!row || row.job_id !== jobId || row.status !== "reserved") return null; | |
| 152 | + this.job(jobId); | |
| 153 | + return toCache(row); | |
| 154 | + } | |
| 155 | + | |
| 156 | + /** Whether entry `id`'s content may still be written (it is reserved by a running job). */ | |
| 157 | + cacheWritable(id: number): boolean { | |
| 158 | + const r = this.sql.exec<{ status: string; job: string | null }>("SELECT c.status, j.status AS job FROM rt_caches c LEFT JOIN jobs j ON j.id = c.job_id WHERE c.id = ?", id).toArray()[0]; | |
| 159 | + return !!r && r.status === "reserved" && r.job === "in_progress"; | |
| 160 | + } | |
| 161 | + | |
| 162 | + /** Bytes of committed entries plus everything staged by uploads in progress. */ | |
| 163 | + private cacheBytes(): number { | |
| 164 | + return this.sql.exec<{ n: number | null }>("SELECT sum(CASE WHEN status = 'ready' THEN size ELSE staged END) AS n FROM rt_caches").one().n ?? 0; | |
| 165 | + } | |
| 166 | + | |
| 167 | + /** Bytes of live artifacts (from both upload paths) plus everything staged by uploads in progress. */ | |
| 168 | + private artifactBytes(): number { | |
| 169 | + const runtime = this.sql.exec<{ n: number | null }>("SELECT sum(CASE WHEN status = 'ready' THEN size ELSE staged END) AS n FROM rt_artifacts WHERE status != 'ready' OR expires_at > ?", now()).one().n ?? 0; | |
| 170 | + const legacy = this.sql.exec<{ n: number | null }>("SELECT sum(size) AS n FROM artifacts").one().n ?? 0; | |
| 171 | + return runtime + legacy; | |
| 172 | + } | |
| 173 | + | |
| 174 | + /** | |
| 175 | + * Accounts for `bytes` an upload is about to stage (a block, a chunk or a whole | |
| 176 | + * blob), before they are written. The upload must belong to a running job and | |
| 177 | + * stay within the repository's budget: for caches, the least recently used | |
| 178 | + * entries are evicted to make room; artifacts have a quota and are refused beyond it. | |
| 179 | + */ | |
| 180 | + async stage(kind: "cache" | "artifact", id: number, bytes: number): Promise<void> { | |
| 181 | + if (!Number.isSafeInteger(bytes) || bytes < 0) throw new HttpError(400, "invalid length"); | |
| 182 | + if (bytes > MAX_UPLOAD_PART) throw new HttpError(413, `a single upload request may carry at most ${MAX_UPLOAD_PART} bytes`); | |
| 183 | + if (kind === "cache") { | |
| 184 | + if (!this.cacheWritable(id)) throw new HttpError(409, "this upload is finished or its job is no longer running"); | |
| 185 | + const own = this.sql.exec<{ staged: number }>("SELECT staged FROM rt_caches WHERE id = ?", id).one().staged; | |
| 186 | + if (own + bytes > CACHE_LIMIT_BYTES) throw new HttpError(413, `a cache entry may be at most ${CACHE_LIMIT_BYTES} bytes`); | |
| 187 | + if (this.cacheBytes() + bytes > CACHE_LIMIT_BYTES) await this.evictCaches(bytes); | |
| 188 | + if (this.cacheBytes() + bytes > CACHE_LIMIT_BYTES) throw new HttpError(413, "uploads in progress use the repository's whole cache budget"); | |
| 189 | + this.sql.exec("UPDATE rt_caches SET staged = staged + ? WHERE id = ?", bytes, id); | |
| 190 | + return; | |
| 191 | + } | |
| 192 | + if (!this.artifactWritable(id)) throw new HttpError(409, "this upload is finished or its job is no longer running"); | |
| 193 | + await this.expireArtifacts(); | |
| 194 | + if (this.artifactBytes() + bytes > this.artifactQuota) throw new HttpError(413, `the repository's artifact storage quota (${this.artifactQuota} bytes) is used up; delete artifacts or wait for them to expire`); | |
| 195 | + this.sql.exec("UPDATE rt_artifacts SET staged = staged + ? WHERE id = ?", bytes, id); | |
| 196 | + } | |
| 197 | + | |
| 198 | + /** Gives back what `stage` counted for a write that then failed, so a failed upload doesn't hold quota. */ | |
| 199 | + unstage(kind: "cache" | "artifact", id: number, bytes: number): void { | |
| 200 | + if (!Number.isSafeInteger(bytes) || bytes <= 0) return; | |
| 201 | + this.sql.exec(`UPDATE ${kind === "cache" ? "rt_caches" : "rt_artifacts"} SET staged = max(0, staged - ?) WHERE id = ?`, bytes, id); | |
| 202 | + } | |
| 203 | + | |
| 204 | + /** Bytes an older runner's artifact upload may still store under the quota. */ | |
| 205 | + async artifactRoom(): Promise<number> { | |
| 206 | + await this.expireArtifacts(); | |
| 207 | + return Math.max(0, this.artifactQuota - this.artifactBytes()); | |
| 208 | + } | |
| 209 | + | |
| 210 | + /** Whether `size` committed bytes for upload `id` fit the budget (evicting caches to make room). */ | |
| 211 | + async checkCommitSize(kind: "cache" | "artifact", id: number, size: number): Promise<void> { | |
| 212 | + if (!Number.isSafeInteger(size) || size < 0) throw new HttpError(400, "invalid size"); | |
| 213 | + if (kind === "cache") { | |
| 214 | + if (size > CACHE_LIMIT_BYTES) throw new HttpError(413, `a cache entry may be at most ${CACHE_LIMIT_BYTES} bytes`); | |
| 215 | + const own = this.sql.exec<{ staged: number }>("SELECT staged FROM rt_caches WHERE id = ?", id).toArray()[0]?.staged ?? 0; | |
| 216 | + if (this.cacheBytes() - own + size > CACHE_LIMIT_BYTES) await this.evictCaches(size - own); | |
| 217 | + if (this.cacheBytes() - own + size > CACHE_LIMIT_BYTES) throw new HttpError(413, "the repository's cache budget is used up by uploads in progress"); | |
| 218 | + return; | |
| 219 | + } | |
| 220 | + const own = this.sql.exec<{ staged: number }>("SELECT staged FROM rt_artifacts WHERE id = ?", id).toArray()[0]?.staged ?? 0; | |
| 221 | + if (this.artifactBytes() - own + size > this.artifactQuota) throw new HttpError(413, `the repository's artifact storage quota (${this.artifactQuota} bytes) is used up`); | |
| 222 | + } | |
| 223 | + | |
| 224 | + /** Commits a reservation; `etag` is the object's, so a later overwrite is detected and refused. */ | |
| 225 | + async cacheCommit(jobId: string, id: number, size: number, etag: string): Promise<CacheEntry> { | |
| 226 | + const r = this.cacheReservation(jobId, { id }); | |
| 227 | + if (!r) throw new HttpError(404, "cache reservation not found"); | |
| 228 | + try { | |
| 229 | + await this.checkCommitSize("cache", id, size); | |
| 230 | + } catch (e) { | |
| 231 | + this.sql.exec("DELETE FROM rt_caches WHERE id = ?", id); | |
| 232 | + await this.dropObjects("cache", [id]); | |
| 233 | + throw e; | |
| 234 | + } | |
| 235 | + const t = now(); | |
| 236 | + this.sql.exec("UPDATE rt_caches SET status = 'ready', size = ?, staged = 0, etag = ?, created_at = ?, last_used = ? WHERE id = ?", size, etag, t, t, id); | |
| 237 | + return { ...r, size, etag, createdAt: t, lastUsed: t }; | |
| 238 | + } | |
| 239 | + | |
| 240 | + cacheGet(id: number): CacheEntry | null { | |
| 241 | + const r = this.sql.exec<CacheRow>("SELECT * FROM rt_caches WHERE id = ? AND status = 'ready'", id).toArray()[0]; | |
| 242 | + return r ? toCache(r) : null; | |
| 243 | + } | |
| 244 | + | |
| 245 | + /** An entry whose object is gone: forget it, so lookups move on to the next match. */ | |
| 246 | + async cacheMissing(id: number): Promise<void> { | |
| 247 | + await this.deleteCacheIds([id]); | |
| 248 | + } | |
| 249 | + | |
| 250 | + /** | |
| 251 | + * Drops entries unused for a week, then the least recently used ones until | |
| 252 | + * committed entries and uploads in progress fit the budget with `room` to spare. | |
| 253 | + */ | |
| 254 | + private async evictCaches(room = 0): Promise<void> { | |
| 255 | + const stale = this.sql.exec<{ id: number }>("SELECT id FROM rt_caches WHERE status = 'ready' AND last_used < ?", new Date(Date.now() - CACHE_IDLE_MS).toISOString()).toArray().map((r) => r.id); | |
| 256 | + let total = this.cacheBytes() + room; | |
| 257 | + const over: number[] = []; | |
| 258 | + for (const r of this.sql.exec<{ id: number; size: number }>("SELECT id, size FROM rt_caches WHERE status = 'ready' ORDER BY last_used, id").toArray()) { | |
| 259 | + if (stale.includes(r.id)) total -= r.size; | |
| 260 | + else if (total > CACHE_LIMIT_BYTES) { | |
| 261 | + over.push(r.id); | |
| 262 | + total -= r.size; | |
| 263 | + } | |
| 264 | + } | |
| 265 | + await this.deleteCacheIds([...stale, ...over]); | |
| 266 | + } | |
| 267 | + | |
| 268 | + private async deleteCacheIds(ids: number[]): Promise<void> { | |
| 269 | + if (!ids.length) return; | |
| 270 | + for (const id of ids) this.sql.exec("DELETE FROM rt_caches WHERE id = ?", id); | |
| 271 | + await this.dropObjects("cache", ids, false); | |
| 272 | + } | |
| 273 | + | |
| 274 | + /** Deletes the objects of dropped rows: the final object and, for unfinished uploads, what they staged. */ | |
| 275 | + private async dropObjects(kind: "cache" | "artifact", ids: number[], staged = true): Promise<void> { | |
| 276 | + const keys = ids.map((id) => finalKey({ repo: this.repoId, kind, id })); | |
| 277 | + for (let i = 0; i < keys.length; i += 1000) await this.bucket.delete(keys.slice(i, i + 1000)); | |
| 278 | + // A page of staged objects per upload; the lifecycle rule on actions-staging/ takes any rest. | |
| 279 | + if (staged) for (const id of ids.slice(0, 20)) await dropStaged(this.bucket, { repo: this.repoId, kind, id }); | |
| 280 | + } | |
| 281 | + | |
| 282 | + /** | |
| 283 | + * When a job ends, its unfinished uploads go: their rows, so they stop counting | |
| 284 | + * against the budgets, and their objects. | |
| 285 | + */ | |
| 286 | + async sweepJob(jobId: string): Promise<void> { | |
| 287 | + const caches = this.sql.exec<{ id: number }>("SELECT id FROM rt_caches WHERE job_id = ? AND status = 'reserved'", jobId).toArray().map((r) => r.id); | |
| 288 | + const artifacts = this.sql.exec<{ id: number }>("SELECT id FROM rt_artifacts WHERE job_id = ? AND status = 'uploading'", jobId).toArray().map((r) => r.id); | |
| 289 | + for (const id of caches) this.sql.exec("DELETE FROM rt_caches WHERE id = ?", id); | |
| 290 | + for (const id of artifacts) this.sql.exec("DELETE FROM rt_artifacts WHERE id = ?", id); | |
| 291 | + if (caches.length) await this.dropObjects("cache", caches); | |
| 292 | + if (artifacts.length) await this.dropObjects("artifact", artifacts); | |
| 293 | + await this.expireArtifacts(); | |
| 294 | + } | |
| 295 | + | |
| 296 | + /** | |
| 297 | + * One step of composing an upload (lib/blob-store.ts), run here so that it gets | |
| 298 | + * an invocation, and subrequests, of its own. It may only touch this upload's | |
| 299 | + * staging area and final object, while the upload is in progress. | |
| 300 | + */ | |
| 301 | + async composeStep(ref: { kind: "cache" | "artifact"; id: number }, step: ComposeStep, contentType?: string): Promise<Array<{ partNumber: number; etag: string }>> { | |
| 302 | + const writable = ref.kind === "cache" ? this.cacheWritable(ref.id) : this.artifactWritable(ref.id); | |
| 303 | + if (!writable) throw new HttpError(409, "this upload is finished or its job is no longer running"); | |
| 304 | + const b = { repo: this.repoId, kind: ref.kind, id: ref.id }; | |
| 305 | + if (!stepKeys(step).every((k) => k === finalKey(b) || k.startsWith(stagingPrefix(b)))) throw new HttpError(400, "the step names objects outside this upload"); | |
| 306 | + return runComposeStep(this.bucket, step, contentType); | |
| 307 | + } | |
| 308 | + | |
| 309 | + /** REST: GET /actions/caches (newest first). */ | |
| 310 | + listCaches(o: { key?: string; ref?: string; sort?: string; direction?: string; page?: number; perPage?: number }): { total: number; caches: CacheEntry[] } { | |
| 311 | + const where = ["status = 'ready'"]; | |
| 312 | + const args: unknown[] = []; | |
| 313 | + if (o.key) { | |
| 314 | + where.push("substr(key, 1, ?) = ?"); | |
| 315 | + args.push(o.key.length, o.key); | |
| 316 | + } | |
| 317 | + if (o.ref) { | |
| 318 | + where.push("scope = ?"); | |
| 319 | + args.push(o.ref); | |
| 320 | + } | |
| 321 | + const col = o.sort === "size_in_bytes" ? "size" : o.sort === "created_at" ? "created_at" : "last_used"; | |
| 322 | + const dir = o.direction === "asc" ? "ASC" : "DESC"; | |
| 323 | + const w = `WHERE ${where.join(" AND ")}`; | |
| 324 | + const total = this.sql.exec<{ n: number }>(`SELECT count(*) AS n FROM rt_caches ${w}`, ...args).one().n; | |
| 325 | + const per = Math.min(Math.max(o.perPage ?? 30, 1), 100); | |
| 326 | + const rows = this.sql.exec<CacheRow>(`SELECT * FROM rt_caches ${w} ORDER BY ${col} ${dir}, id ${dir} LIMIT ? OFFSET ?`, ...args, per, (Math.max(o.page ?? 1, 1) - 1) * per).toArray(); | |
| 327 | + return { total, caches: rows.map(toCache) }; | |
| 328 | + } | |
| 329 | + | |
| 330 | + /** REST: DELETE /actions/caches?key=&ref= or /actions/caches/:id. */ | |
| 331 | + async deleteCaches(o: { id?: number; key?: string; ref?: string }): Promise<CacheEntry[]> { | |
| 332 | + const rows = | |
| 333 | + o.id !== undefined | |
| 334 | + ? this.sql.exec<CacheRow>("SELECT * FROM rt_caches WHERE id = ? AND status = 'ready'", o.id).toArray() | |
| 335 | + : this.sql.exec<CacheRow>("SELECT * FROM rt_caches WHERE key = ? AND status = 'ready'" + (o.ref ? " AND scope = ?" : ""), ...(o.ref ? [o.key ?? "", o.ref] : [o.key ?? ""])).toArray(); | |
| 336 | + await this.deleteCacheIds(rows.map((r) => r.id)); | |
| 337 | + return rows.map(toCache); | |
| 338 | + } | |
| 339 | + | |
| 340 | + cacheUsage(): { count: number; size: number } { | |
| 341 | + const r = this.sql.exec<{ n: number; s: number | null }>("SELECT count(*) AS n, sum(size) AS s FROM rt_caches WHERE status = 'ready'").one(); | |
| 342 | + return { count: r.n, size: r.s ?? 0 }; | |
| 343 | + } | |
| 344 | + | |
| 345 | + // ---------------------------------------------------------------- artifacts | |
| 346 | + | |
| 347 | + private toArtifact = (r: ArtifactRow): ArtifactEntry => { | |
| 348 | + const run = this.sql.exec<{ n: number; ref: string; sha: string }>("SELECT run_number AS n, ref, sha FROM runs WHERE id = ?", r.run_id).toArray()[0]; | |
| 349 | + return { | |
| 350 | + id: r.id, | |
| 351 | + runId: r.run_id, | |
| 352 | + runNumber: run?.n ?? 0, | |
| 353 | + headBranch: (run?.ref ?? "").replace(/^refs\/(heads|tags)\//, ""), | |
| 354 | + headSha: run?.sha ?? "", | |
| 355 | + jobId: r.job_id, | |
| 356 | + name: r.name, | |
| 357 | + size: r.size, | |
| 358 | + digest: r.digest, | |
| 359 | + mimeType: r.mime_type, | |
| 360 | + createdAt: r.created_at, | |
| 361 | + updatedAt: r.updated_at, | |
| 362 | + expiresAt: r.expires_at, | |
| 363 | + expired: r.status === "expired" || r.expires_at <= now(), | |
| 364 | + etag: r.etag, | |
| 365 | + }; | |
| 366 | + }; | |
| 367 | + | |
| 368 | + async artifactCreate(jobId: string, name: string, mimeType: string | undefined, expiresAt: string | undefined): Promise<{ id: number }> { | |
| 369 | + const { runId } = this.job(jobId); | |
| 370 | + if (!name || name.length > 256 || /[":<>|*?\\/\r\n]/.test(name)) throw new HttpError(400, `invalid artifact name: ${name}`); | |
| 371 | + await this.expireArtifacts(); | |
| 372 | + const existing = this.sql.exec<ArtifactRow>("SELECT * FROM rt_artifacts WHERE run_id = ? AND name = ?", runId, name).toArray(); | |
| 373 | + if (existing.some((a) => a.status === "ready")) throw new HttpError(409, `an artifact with this name already exists on the workflow run`); | |
| 374 | + // Unfinished uploads (a crashed earlier attempt) are replaced, objects and all. | |
| 375 | + for (const a of existing) this.sql.exec("DELETE FROM rt_artifacts WHERE id = ?", a.id); | |
| 376 | + if (existing.length) await this.dropObjects("artifact", existing.map((a) => a.id)); | |
| 377 | + const maxExpiry = Date.now() + MAX_RETENTION_DAYS * 86400_000; | |
| 378 | + const requested = expiresAt ? Date.parse(expiresAt) : NaN; | |
| 379 | + const expiry = new Date(Number.isFinite(requested) ? Math.min(Math.max(requested, Date.now() + 86400_000), maxExpiry) : maxExpiry).toISOString(); | |
| 380 | + const t = now(); | |
| 381 | + this.sql.exec( | |
| 382 | + "INSERT INTO rt_artifacts (run_id, job_id, name, size, mime_type, status, created_at, updated_at, expires_at) VALUES (?, ?, ?, 0, ?, 'uploading', ?, ?, ?)", | |
| 383 | + runId, | |
| 384 | + jobId, | |
| 385 | + name, | |
| 386 | + mimeType || "application/zip", | |
| 387 | + t, | |
| 388 | + t, | |
| 389 | + expiry, | |
| 390 | + ); | |
| 391 | + return { id: this.sql.exec<{ id: number }>("SELECT last_insert_rowid() AS id").one().id }; | |
| 392 | + } | |
| 393 | + | |
| 394 | + /** The job's unfinished upload of artifact `name`. */ | |
| 395 | + artifactUpload(jobId: string, name: string): ArtifactEntry | null { | |
| 396 | + const { runId } = this.job(jobId); | |
| 397 | + const r = this.sql.exec<ArtifactRow>("SELECT * FROM rt_artifacts WHERE run_id = ? AND name = ? AND job_id = ? AND status = 'uploading' ORDER BY id DESC LIMIT 1", runId, name, jobId).toArray()[0]; | |
| 398 | + return r ? this.toArtifact(r) : null; | |
| 399 | + } | |
| 400 | + | |
| 401 | + artifactWritable(id: number): boolean { | |
| 402 | + const r = this.sql.exec<{ status: string; job: string | null }>("SELECT a.status, j.status AS job FROM rt_artifacts a LEFT JOIN jobs j ON j.id = a.job_id WHERE a.id = ?", id).toArray()[0]; | |
| 403 | + return !!r && r.status === "uploading" && r.job === "in_progress"; | |
| 404 | + } | |
| 405 | + | |
| 406 | + /** Finishes an upload; `etag` is the object's, so a later overwrite is detected and refused. */ | |
| 407 | + async artifactFinalize(jobId: string, name: string, size: number, digest: string | undefined, etag: string): Promise<ArtifactEntry> { | |
| 408 | + const a = this.artifactUpload(jobId, name); | |
| 409 | + if (!a) throw new HttpError(404, `no upload of artifact '${name}' in progress`); | |
| 410 | + try { | |
| 411 | + await this.checkCommitSize("artifact", a.id, size); | |
| 412 | + } catch (e) { | |
| 413 | + this.sql.exec("DELETE FROM rt_artifacts WHERE id = ?", a.id); | |
| 414 | + await this.dropObjects("artifact", [a.id]); | |
| 415 | + throw e; | |
| 416 | + } | |
| 417 | + this.sql.exec("UPDATE rt_artifacts SET status = 'ready', size = ?, staged = 0, digest = ?, etag = ?, updated_at = ? WHERE id = ?", size, digest ?? null, etag, now(), a.id); | |
| 418 | + return this.artifactGet(a.id)!; | |
| 419 | + } | |
| 420 | + | |
| 421 | + /** A finished artifact; with `includeExpired` (the REST API), also one past its retention. */ | |
| 422 | + artifactGet(id: number, o: { includeExpired?: boolean } = {}): ArtifactEntry | null { | |
| 423 | + const r = o.includeExpired | |
| 424 | + ? this.sql.exec<ArtifactRow>("SELECT * FROM rt_artifacts WHERE id = ? AND status IN ('ready', 'expired')", id).toArray()[0] | |
| 425 | + : this.sql.exec<ArtifactRow>("SELECT * FROM rt_artifacts WHERE id = ? AND status = 'ready' AND expires_at > ?", id, now()).toArray()[0]; | |
| 426 | + return r ? this.toArtifact(r) : null; | |
| 427 | + } | |
| 428 | + | |
| 429 | + /** | |
| 430 | + * Finished, unexpired artifacts of a run (latest first), optionally by name or id; | |
| 431 | + * with `includeExpired` (the REST API), expired ones too. | |
| 432 | + * A numeric run id (github.run_id, the run number) covers all of the run's attempts. | |
| 433 | + */ | |
| 434 | + runArtifacts(runId: string, filter: { name?: string; id?: number; includeExpired?: boolean } = {}): ArtifactEntry[] { | |
| 435 | + const numeric = /^\d+$/.test(runId); | |
| 436 | + const where = [numeric ? "run_id IN (SELECT id FROM runs WHERE run_number = ?)" : "run_id = ?", ...(filter.includeExpired ? ["status IN ('ready', 'expired')"] : ["status = 'ready'", "expires_at > ?"])]; | |
| 437 | + const args: unknown[] = [numeric ? Number(runId) : runId, ...(filter.includeExpired ? [] : [now()])]; | |
| 438 | + if (filter.name !== undefined) { | |
| 439 | + where.push("name = ?"); | |
| 440 | + args.push(filter.name); | |
| 441 | + } | |
| 442 | + if (filter.id !== undefined) { | |
| 443 | + where.push("id = ?"); | |
| 444 | + args.push(filter.id); | |
| 445 | + } | |
| 446 | + return this.sql.exec<ArtifactRow>(`SELECT * FROM rt_artifacts WHERE ${where.join(" AND ")} ORDER BY id DESC`, ...args).toArray().map(this.toArtifact); | |
| 447 | + } | |
| 448 | + | |
| 449 | + /** The artifacts of a running job's own run (the toolkit's ListArtifacts and GetSignedArtifactURL). */ | |
| 450 | + jobArtifacts(jobId: string, filter: { name?: string; id?: number } = {}): ArtifactEntry[] { | |
| 451 | + return this.runArtifacts(this.job(jobId).runId, filter); | |
| 452 | + } | |
| 453 | + | |
| 454 | + /** REST: GET /actions/artifacts, expired artifacts included (as GitHub lists them). */ | |
| 455 | + repoArtifacts(o: { name?: string; page?: number; perPage?: number }): { total: number; artifacts: ArtifactEntry[] } { | |
| 456 | + const where = ["status IN ('ready', 'expired')"]; | |
| 457 | + const args: unknown[] = []; | |
| 458 | + if (o.name) { | |
| 459 | + where.push("name = ?"); | |
| 460 | + args.push(o.name); | |
| 461 | + } | |
| 462 | + const w = `WHERE ${where.join(" AND ")}`; | |
| 463 | + const total = this.sql.exec<{ n: number }>(`SELECT count(*) AS n FROM rt_artifacts ${w}`, ...args).one().n; | |
| 464 | + const per = Math.min(Math.max(o.perPage ?? 30, 1), 100); | |
| 465 | + const rows = this.sql.exec<ArtifactRow>(`SELECT * FROM rt_artifacts ${w} ORDER BY id DESC LIMIT ? OFFSET ?`, ...args, per, (Math.max(o.page ?? 1, 1) - 1) * per).toArray(); | |
| 466 | + return { total, artifacts: rows.map(this.toArtifact) }; | |
| 467 | + } | |
| 468 | + | |
| 469 | + /** Deletes an artifact (REST, or the toolkit's DeleteArtifact for `overwrite: true`). */ | |
| 470 | + async artifactDelete(id: number): Promise<ArtifactEntry | null> { | |
| 471 | + const a = this.artifactGet(id, { includeExpired: true }); | |
| 472 | + this.sql.exec("DELETE FROM rt_artifacts WHERE id = ?", id); | |
| 473 | + await this.dropObjects("artifact", [id], false); | |
| 474 | + return a; | |
| 475 | + } | |
| 476 | + | |
| 477 | + /** The job's own run's artifact `name`, for the toolkit's DeleteArtifact. */ | |
| 478 | + async artifactDeleteByName(jobId: string, name: string): Promise<ArtifactEntry | null> { | |
| 479 | + const { runId } = this.job(jobId); | |
| 480 | + const a = this.runArtifacts(runId, { name })[0]; | |
| 481 | + return a ? this.artifactDelete(a.id) : null; | |
| 482 | + } | |
| 483 | + | |
| 484 | + /** | |
| 485 | + * Artifacts past their retention: the content is deleted and the artifact stays | |
| 486 | + * listed as expired (GitHub's `expired: true`; downloads get 410) for a while; | |
| 487 | + * unfinished uploads past it go entirely. Runs when uploads start and jobs end. | |
| 488 | + */ | |
| 489 | + async expireArtifacts(): Promise<void> { | |
| 490 | + const t = now(); | |
| 491 | + const due = this.sql.exec<{ id: number; status: string }>("SELECT id, status FROM rt_artifacts WHERE expires_at <= ? AND status != 'expired' LIMIT 500", t).toArray(); | |
| 492 | + const ready = due.filter((a) => a.status === "ready").map((a) => a.id); | |
| 493 | + const unfinished = due.filter((a) => a.status !== "ready").map((a) => a.id); | |
| 494 | + for (const id of ready) this.sql.exec("UPDATE rt_artifacts SET status = 'expired', staged = 0 WHERE id = ?", id); | |
| 495 | + for (const id of unfinished) this.sql.exec("DELETE FROM rt_artifacts WHERE id = ?", id); | |
| 496 | + if (ready.length) await this.dropObjects("artifact", ready, false); | |
| 497 | + if (unfinished.length) await this.dropObjects("artifact", unfinished); | |
| 498 | + const forgotten = new Date(Date.now() - EXPIRED_LISTED_DAYS * 86400_000).toISOString(); | |
| 499 | + this.sql.exec("DELETE FROM rt_artifacts WHERE status = 'expired' AND expires_at <= ?", forgotten); | |
| 500 | + } | |
| 501 | +} |
modifiedapps/diffinery/src/repo/schema.ts+32−0View file
| @@ -69,6 +69,22 @@ | ||
| 69 | 69 | ); |
| 70 | 70 | CREATE TABLE IF NOT EXISTS sandboxes (id TEXT PRIMARY KEY, secret TEXT NOT NULL, created_by TEXT NOT NULL, ref TEXT NOT NULL, created_at TEXT NOT NULL, status TEXT NOT NULL); |
| 71 | 71 | |
| 72 | +-- Actions runtime services (repo/runtime.ts): the cache and artifact services the | |
| 73 | +-- official toolkit talks to, with numeric ids like GitHub's. Content lives in R2. | |
| 74 | +CREATE TABLE IF NOT EXISTS rt_caches ( | |
| 75 | + id INTEGER PRIMARY KEY AUTOINCREMENT, key TEXT NOT NULL, version TEXT NOT NULL, scope TEXT NOT NULL, | |
| 76 | + size INTEGER NOT NULL DEFAULT 0, status TEXT NOT NULL, job_id TEXT, created_at TEXT NOT NULL, last_used TEXT NOT NULL, | |
| 77 | + UNIQUE(key, version, scope) | |
| 78 | +); | |
| 79 | +CREATE INDEX IF NOT EXISTS rt_caches_used ON rt_caches(last_used); | |
| 80 | +CREATE TABLE IF NOT EXISTS rt_artifacts ( | |
| 81 | + id INTEGER PRIMARY KEY AUTOINCREMENT, run_id TEXT NOT NULL, job_id TEXT NOT NULL, name TEXT NOT NULL, | |
| 82 | + size INTEGER NOT NULL DEFAULT 0, digest TEXT, mime_type TEXT NOT NULL DEFAULT 'application/zip', status TEXT NOT NULL, | |
| 83 | + created_at TEXT NOT NULL, updated_at TEXT NOT NULL, expires_at TEXT NOT NULL | |
| 84 | +); | |
| 85 | +CREATE INDEX IF NOT EXISTS rt_artifacts_run ON rt_artifacts(run_id, name); | |
| 86 | +-- What a job's GITHUB_TOKEN may do, from the workflow's permissions (repo/runtime.ts). | |
| 87 | +CREATE TABLE IF NOT EXISTS job_grants (job_id TEXT PRIMARY KEY, role TEXT NOT NULL, permissions TEXT NOT NULL); | |
| 72 | 88 | -- GitHub-compatible extras (see ./releases.ts, ./statuses.ts, ./deployments.ts, ./social.ts). |
| 73 | 89 | CREATE TABLE IF NOT EXISTS releases ( |
| 74 | 90 | id INTEGER PRIMARY KEY AUTOINCREMENT, tag_name TEXT NOT NULL, target_commitish TEXT NOT NULL, name TEXT NOT NULL DEFAULT '', |
| @@ -115,6 +131,8 @@ | ||
| 115 | 131 | actor_kind TEXT NOT NULL, detail TEXT NOT NULL DEFAULT '{}', created_at TEXT NOT NULL |
| 116 | 132 | ); |
| 117 | 133 | CREATE INDEX IF NOT EXISTS issue_events_number ON issue_events(number); |
| 134 | +-- Who last moved each ref: tags made by agents never match environment tag patterns | |
| 135 | +-- (./deployments.ts), and runs of code an agent pushed act as the agent (./token-policy.ts). | |
| 118 | 136 | CREATE TABLE IF NOT EXISTS ref_actors (ref TEXT PRIMARY KEY, actor_id TEXT NOT NULL, actor_kind TEXT NOT NULL, updated_at TEXT NOT NULL); |
| 119 | 137 | `; |
| 120 | 138 | |
| @@ -123,11 +141,25 @@ | ||
| 123 | 141 | // The commit a run reports its checks on: the PR head for pull_request runs, whose |
| 124 | 142 | // GITHUB_SHA is the test merge commit. Null means the run's own sha. |
| 125 | 143 | ["runs", "head_sha", "TEXT"], |
| 144 | + // 1 when the code a run executes came from an agent (repo/actions.ts agentOrigin): its jobs act as an agent. | |
| 145 | + ["runs", "agent_origin", "INTEGER NOT NULL DEFAULT 0"], | |
| 146 | + // The repository a pull request's head branch lives in, when it is not this one (a fork). | |
| 147 | + ["pulls", "head_repo_id", "TEXT"], | |
| 148 | + // Bytes an upload in progress has staged (counted against the budgets), and the | |
| 149 | + // committed object's etag (a later overwrite of the object is refused). | |
| 150 | + ["rt_caches", "staged", "INTEGER NOT NULL DEFAULT 0"], | |
| 151 | + ["rt_caches", "etag", "TEXT"], | |
| 152 | + ["rt_artifacts", "staged", "INTEGER NOT NULL DEFAULT 0"], | |
| 153 | + ["rt_artifacts", "etag", "TEXT"], | |
| 126 | 154 | // Who created a check run over the REST API (null for Diffinery Actions), and GitHub's numeric id. |
| 127 | 155 | ["checks", "creator_id", "TEXT"], |
| 128 | 156 | ["checks", "creator_login", "TEXT"], |
| 129 | 157 | ["checks", "creator_kind", "TEXT"], |
| 130 | 158 | ["checks", "seq", "INTEGER"], |
| 159 | + // Per matrix leg: job-level continue-on-error, and the job-level concurrency group with its cancel-in-progress. | |
| 160 | + ["jobs", "continue_on_error", "INTEGER NOT NULL DEFAULT 0"], | |
| 161 | + ["jobs", "concurrency_group", "TEXT"], | |
| 162 | + ["jobs", "concurrency_cancel", "INTEGER NOT NULL DEFAULT 0"], | |
| 131 | 163 | ]; |
| 132 | 164 | |
| 133 | 165 | export function migrateRepoSchema(sql: SqlStorage): void { |
addedapps/diffinery/src/repo/token-policy.ts+141−0View file
| @@ -1,0 +1,141 @@ | ||
| 1 | +import { ZERO_OID, diffTrees, mergeBase, peelToCommit, readCommit, resolvePath } from "@diffinery/git-core"; | |
| 2 | +import { HttpError, now } from "../lib/util.js"; | |
| 3 | +import type { RepoDO } from "./RepoDO.js"; | |
| 4 | +import type { Caller } from "./types.js"; | |
| 5 | + | |
| 6 | +/** | |
| 7 | + * What a workflow's GITHUB_TOKEN is trusted with, beyond its `permissions:` | |
| 8 | + * (lib/job-permissions.ts): | |
| 9 | + * | |
| 10 | + * - The repository's Actions settings, as on GitHub: the default token | |
| 11 | + * permissions (read-only for repositories created from now on, read and write | |
| 12 | + * for older ones) and whether workflows may create and approve pull requests. | |
| 13 | + * - Whether a run's code came from an agent: the agent pushed it, opened the | |
| 14 | + * pull request, or pushed the branch last. Such runs act as an agent, so the | |
| 15 | + * agent guardrails (no pushes or merges to protected and default branches, | |
| 16 | + * advisory reviews) hold for code an agent wrote, whoever triggered the run. | |
| 17 | + * - Workflow files: a job token never creates or changes them (GitHub's | |
| 18 | + * `workflows` permission, which GITHUB_TOKEN cannot have). | |
| 19 | + */ | |
| 20 | + | |
| 21 | +export type WorkflowSettings = { defaultPermissions: "read" | "write"; canApprovePullRequests: boolean }; | |
| 22 | + | |
| 23 | +const SETTINGS_KEY = "actions_workflow_settings"; | |
| 24 | +/** Repositories that predate the setting keep GitHub's old default. */ | |
| 25 | +const LEGACY: WorkflowSettings = { defaultPermissions: "write", canApprovePullRequests: false }; | |
| 26 | +const NEW_REPO: WorkflowSettings = { defaultPermissions: "read", canApprovePullRequests: false }; | |
| 27 | + | |
| 28 | +export const WORKFLOW_DIRS = [".github/workflows", ".diffinery/workflows"]; | |
| 29 | + | |
| 30 | +export const isJobCaller = (c: { id: string }) => c.id.startsWith("job:"); | |
| 31 | + | |
| 32 | +export class TokenPolicy { | |
| 33 | + constructor(private repo: RepoDO) {} | |
| 34 | + | |
| 35 | + private get sql() { | |
| 36 | + return this.repo.sql; | |
| 37 | + } | |
| 38 | + | |
| 39 | + // ---------------------------------------------------------------- settings | |
| 40 | + | |
| 41 | + /** Called when a repository is created: it starts with GitHub's current (read-only) default. */ | |
| 42 | + initNewRepo(): void { | |
| 43 | + this.sql.exec("INSERT OR IGNORE INTO meta (key, value) VALUES (?, ?)", SETTINGS_KEY, JSON.stringify(NEW_REPO)); | |
| 44 | + } | |
| 45 | + | |
| 46 | + settings(): WorkflowSettings { | |
| 47 | + const row = this.sql.exec<{ value: string }>("SELECT value FROM meta WHERE key = ?", SETTINGS_KEY).toArray()[0]; | |
| 48 | + return row ? { ...LEGACY, ...(JSON.parse(row.value) as Partial<WorkflowSettings>) } : { ...LEGACY }; | |
| 49 | + } | |
| 50 | + | |
| 51 | + setSettings(patch: Partial<WorkflowSettings>, caller: Caller): WorkflowSettings { | |
| 52 | + if (caller.role !== "admin") throw new HttpError(403, "admin access required"); | |
| 53 | + if (caller.kind !== "user") throw new HttpError(403, "only people can change what workflows are allowed to do"); | |
| 54 | + if (patch.defaultPermissions !== undefined && patch.defaultPermissions !== "read" && patch.defaultPermissions !== "write") throw new HttpError(422, "default_workflow_permissions must be read or write"); | |
| 55 | + if (patch.canApprovePullRequests !== undefined && typeof patch.canApprovePullRequests !== "boolean") throw new HttpError(422, "can_approve_pull_request_reviews must be a boolean"); | |
| 56 | + const next = { ...this.settings(), ...Object.fromEntries(Object.entries(patch).filter(([, v]) => v !== undefined)) } as WorkflowSettings; | |
| 57 | + this.sql.exec("INSERT OR REPLACE INTO meta (key, value) VALUES (?, ?)", SETTINGS_KEY, JSON.stringify(next)); | |
| 58 | + const m = this.repo.meta; | |
| 59 | + this.repo.emit({ type: "audit", action: "repo.actions_workflow_settings", actor: { id: caller.id, login: caller.login, kind: caller.kind, onBehalfOf: caller.onBehalfOf }, target: `${m.owner}/${m.name}`, detail: { ...next }, at: now() }); | |
| 60 | + return next; | |
| 61 | + } | |
| 62 | + | |
| 63 | + /** Workflows creating or approving pull requests is a setting, off by default (GitHub's message). */ | |
| 64 | + assertMayCreateOrApprove(caller: Caller, what: "create" | "approve"): void { | |
| 65 | + if (!isJobCaller(caller) || this.settings().canApprovePullRequests) return; | |
| 66 | + throw new HttpError(403, `GitHub Actions is not permitted to ${what === "create" ? "create" : "approve"} pull requests. An admin can allow it in the repository's Actions settings ("Allow GitHub Actions to create and approve pull requests").`); | |
| 67 | + } | |
| 68 | + | |
| 69 | + // ---------------------------------------------------------------- where code came from | |
| 70 | + | |
| 71 | + /** Records who moved `ref` (pushes, merges, API commits). */ | |
| 72 | + recordPush(ref: string, newOid: string, caller: Caller): void { | |
| 73 | + if (newOid === ZERO_OID) this.sql.exec("DELETE FROM ref_actors WHERE ref = ?", ref); | |
| 74 | + else this.sql.exec("INSERT OR REPLACE INTO ref_actors (ref, actor_id, actor_kind, updated_at) VALUES (?, ?, ?, ?)", ref, caller.id, caller.kind, now()); | |
| 75 | + } | |
| 76 | + | |
| 77 | + lastPusherKind(ref: string): string | null { | |
| 78 | + return this.sql.exec<{ actor_kind: string }>("SELECT actor_kind FROM ref_actors WHERE ref = ?", ref).toArray()[0]?.actor_kind ?? null; | |
| 79 | + } | |
| 80 | + | |
| 81 | + /** | |
| 82 | + * Whether a new run's code came from an agent: the trigger was an agent, the run | |
| 83 | + * re-runs one that did (`inherited`), the branch it runs was last pushed by an | |
| 84 | + * agent, or, for pull request events, an agent opened the pull request or last | |
| 85 | + * pushed its head branch. | |
| 86 | + */ | |
| 87 | + agentOrigin(o: { caller: { kind: string }; ref: string; prNumbers?: number[]; inherited?: boolean }): boolean { | |
| 88 | + if (o.inherited || o.caller.kind === "agent") return true; | |
| 89 | + if (o.ref.startsWith("refs/heads/") && this.lastPusherKind(o.ref) === "agent") return true; | |
| 90 | + for (const n of o.prNumbers ?? []) { | |
| 91 | + const pr = this.sql.exec<{ author_kind: string; head_ref: string }>("SELECT i.author_kind, p.head_ref FROM issues i JOIN pulls p ON p.number = i.number WHERE i.number = ?", n).toArray()[0]; | |
| 92 | + if (!pr) continue; | |
| 93 | + if (pr.author_kind === "agent" || this.lastPusherKind(`refs/heads/${pr.head_ref}`) === "agent") return true; | |
| 94 | + } | |
| 95 | + return false; | |
| 96 | + } | |
| 97 | + | |
| 98 | + /** A pull request whose head is in another repository (a fork), from the pull request's own row. */ | |
| 99 | + isForkPull(number: number | null): boolean { | |
| 100 | + if (number === null) return false; | |
| 101 | + const r = this.sql.exec<{ head_repo_id: string | null }>("SELECT head_repo_id FROM pulls WHERE number = ?", number).toArray()[0]; | |
| 102 | + return !!r?.head_repo_id && r.head_repo_id !== this.repo.meta.repoId; | |
| 103 | + } | |
| 104 | + | |
| 105 | + // ---------------------------------------------------------------- workflow files | |
| 106 | + | |
| 107 | + private async workflowDir(commit: string | null, dir: string): Promise<string | null> { | |
| 108 | + if (!commit) return null; | |
| 109 | + const c = await readCommit(this.repo.store, await peelToCommit(this.repo.store, commit)); | |
| 110 | + const e = await resolvePath(this.repo.store, c.tree, dir); | |
| 111 | + return e?.oid ?? null; | |
| 112 | + } | |
| 113 | + | |
| 114 | + /** | |
| 115 | + * The first workflow file that moving a ref from `oldOid` to `newOid` creates, | |
| 116 | + * changes or deletes. A new ref is compared with where it branched off the | |
| 117 | + * default branch. | |
| 118 | + */ | |
| 119 | + async changedWorkflow(oldOid: string, newOid: string): Promise<string | null> { | |
| 120 | + if (newOid === ZERO_OID) return null; | |
| 121 | + let base: string | null = oldOid === ZERO_OID ? null : oldOid; | |
| 122 | + if (!base) { | |
| 123 | + const head = this.repo.getRef(this.repo.headRef); | |
| 124 | + base = head ? await mergeBase(this.repo.store, head, await peelToCommit(this.repo.store, newOid)) : null; | |
| 125 | + } | |
| 126 | + for (const dir of WORKFLOW_DIRS) { | |
| 127 | + const [a, b] = await Promise.all([this.workflowDir(base, dir), this.workflowDir(newOid, dir)]); | |
| 128 | + if (a === b) continue; | |
| 129 | + const changes = await diffTrees(this.repo.store, a, b, dir).catch(() => []); | |
| 130 | + return changes[0]?.path ?? dir; | |
| 131 | + } | |
| 132 | + return null; | |
| 133 | + } | |
| 134 | + | |
| 135 | + /** GitHub's refusal for a job token that would add or change a workflow file. */ | |
| 136 | + async refuseWorkflowEdit(caller: Caller, oldOid: string, newOid: string): Promise<string | null> { | |
| 137 | + if (!isJobCaller(caller)) return null; | |
| 138 | + const path = await this.changedWorkflow(oldOid, newOid); | |
| 139 | + return path ? `refusing to allow a GitHub App to create or update workflow \`${path}\` without \`workflows\` permission` : null; | |
| 140 | + } | |
| 141 | +} |
modifiedapps/diffinery/src/repo/types.ts+2−0View file
| @@ -7,6 +7,8 @@ | ||
| 7 | 7 | /** The authenticated caller, with their effective role on this repository. */ |
| 8 | 8 | export interface Caller extends Actor { |
| 9 | 9 | role: Role; |
| 10 | + /** A workflow job token's effective `permissions:` (lib/job-permissions.ts); absent for everyone else. */ | |
| 11 | + permissions?: Record<string, "none" | "read" | "write">; | |
| 10 | 12 | } |
| 11 | 13 | |
| 12 | 14 | export interface RepoMeta { |
modifiedapps/diffinery/src/runner/workflow.ts+21−19View file
| @@ -1,5 +1,6 @@ | ||
| 1 | 1 | import { WorkflowEntrypoint, type WorkflowEvent, type WorkflowStep } from "cloudflare:workers"; |
| 2 | 2 | import type { Env } from "../env.js"; |
| 3 | +import { graphFromLevels, runGraph, type JobNode } from "../repo/job-graph.js"; | |
| 3 | 4 | |
| 4 | 5 | export interface RunParams { |
| 5 | 6 | repoId: string; |
| @@ -7,9 +8,11 @@ | ||
| 7 | 8 | } |
| 8 | 9 | |
| 9 | 10 | /** |
| 10 | - * Durable orchestration of one workflow run: resolves the job graph level by | |
| 11 | - * level, starts each job's containers, and waits for the runner to report | |
| 12 | - * completion. Retries, sleeps and waits survive restarts of the platform. | |
| 11 | + * Durable orchestration of one workflow run: walks the job graph, starting each | |
| 12 | + * job's containers as soon as the jobs it needs have finished, and waits for the | |
| 13 | + * runner to report completion. Retries, sleeps and waits survive restarts of the | |
| 14 | + * platform. Each job costs a fixed number of steps (start, then a status check | |
| 15 | + * and one wait per matrix leg), however long it runs. | |
| 13 | 16 | */ |
| 14 | 17 | export class RunWorkflow extends WorkflowEntrypoint<Env, RunParams> { |
| 15 | 18 | override async run(event: WorkflowEvent<RunParams>, step: WorkflowStep): Promise<unknown> { |
| @@ -27,22 +30,21 @@ | ||
| 27 | 30 | return step.do("finish", () => repo().finishRun(runId)); |
| 28 | 31 | } |
| 29 | 32 | |
| 30 | - for (const [levelIndex, level] of plan.levels.entries()) { | |
| 31 | - const started = await Promise.all(level.map((key) => step.do(`start ${key}`, { retries: { limit: 3, delay: "10 seconds", backoff: "exponential" } }, () => repo().startJobKey(runId, key)))); | |
| 32 | - const jobIds = started.flatMap((s) => s.jobIds); | |
| 33 | - await Promise.all( | |
| 34 | - jobIds.map(async (jobId) => { | |
| 35 | - const status = await step.do(`job ${jobId} info`, () => repo().jobStatus(jobId)); | |
| 36 | - if (status === "completed") return; | |
| 37 | - try { | |
| 38 | - await step.waitForEvent(`wait ${jobId}`, { type: `job-${jobId}`, timeout: "6 hours" }); | |
| 39 | - } catch { | |
| 40 | - await step.do(`timeout ${jobId}`, () => repo().jobTimedOut(jobId)); | |
| 41 | - } | |
| 42 | - }), | |
| 43 | - ); | |
| 44 | - void levelIndex; | |
| 45 | - } | |
| 33 | + const waitFor = async (jobId: string) => { | |
| 34 | + const status = await step.do(`job ${jobId} info`, () => repo().jobStatus(jobId)); | |
| 35 | + if (status === "completed") return; | |
| 36 | + try { | |
| 37 | + await step.waitForEvent(`wait ${jobId}`, { type: `job-${jobId}`, timeout: "6 hours" }); | |
| 38 | + } catch { | |
| 39 | + await step.do(`timeout ${jobId}`, () => repo().jobTimedOut(jobId)); | |
| 40 | + } | |
| 41 | + }; | |
| 42 | + // Runs planned before plans carried a graph replay with their recorded levels. | |
| 43 | + const legacy = plan as { jobs?: JobNode[]; levels?: string[][] }; | |
| 44 | + await runGraph(legacy.jobs ?? graphFromLevels(legacy.levels ?? []), async (key) => { | |
| 45 | + const started = await step.do(`start ${key}`, { retries: { limit: 3, delay: "10 seconds", backoff: "exponential" } }, () => repo().startJobKey(runId, key)); | |
| 46 | + await Promise.all(started.jobIds.map(waitFor)); | |
| 47 | + }); | |
| 46 | 48 | return step.do("finish", () => repo().finishRun(runId)); |
| 47 | 49 | } |
| 48 | 50 | } |
modifiedapps/diffinery/src/web/collab.tsx+4−4View file
| @@ -677,7 +677,7 @@ | ||
| 677 | 677 | : m.state === "draft" |
| 678 | 678 | ? "This pull request is still a work in progress" |
| 679 | 679 | : "Merging is blocked"; |
| 680 | - const bypass = !m.mergeable && caller.role === "admin" && caller.kind === "user" && m.conflicts.length === 0 && !p.draft; | |
| 680 | + const bypass = !m.mergeable && caller.role === "admin" && caller.kind === "user" && !m.enforceAdmins && m.conflicts.length === 0 && !p.draft; | |
| 681 | 681 | const showChecks = checks.length ? checks : m.checks.map((ch) => ({ name: ch.name, status: ch.state === "queued" || ch.state === "in_progress" ? ch.state : "completed", conclusion: ch.state, runId: null, jobId: null, output: null }) as unknown as Check); |
| 682 | 682 | const failing = showChecks.filter((ch) => ch.status === "completed" && ch.conclusion !== "success" && ch.conclusion !== "skipped" && ch.conclusion !== "neutral").length; |
| 683 | 683 | const pending = showChecks.filter((ch) => ch.status !== "completed").length; |
| @@ -744,7 +744,7 @@ | ||
| 744 | 744 | ) : null} |
| 745 | 745 | </div> |
| 746 | 746 | {m.mergeQueue ? ( |
| 747 | - <QueueBox x={x} m={m} canAct={isAccount && atLeast(caller.role, "write") && !x.l.repo.archived} adminBypass={caller.role === "admin" && caller.kind === "user" && m.conflicts.length === 0 && !p.draft} /> | |
| 747 | + <QueueBox x={x} m={m} canAct={isAccount && atLeast(caller.role, "write") && !x.l.repo.archived} adminBypass={caller.role === "admin" && caller.kind === "user" && !m.enforceAdmins && m.conflicts.length === 0 && !p.draft} /> | |
| 748 | 748 | ) : isAccount && atLeast(caller.role, "write") && !x.l.repo.archived ? ( |
| 749 | 749 | <div> |
| 750 | 750 | <Form action={`${base}/pull/${i.number}/merge`} t={csrfToken(x.c)} class="row" confirm={bypass ? "Merge without meeting the branch requirements?" : undefined}> |
| @@ -753,9 +753,9 @@ | ||
| 753 | 753 | <KSelect |
| 754 | 754 | name="method" |
| 755 | 755 | label="Merge method" |
| 756 | - value="merge" | |
| 756 | + value={m.linearHistory ? "squash" : "merge"} | |
| 757 | 757 | options={[ |
| 758 | - { value: "merge", label: "Create a merge commit" }, | |
| 758 | + ...(m.linearHistory ? [] : [{ value: "merge", label: "Create a merge commit" }]), | |
| 759 | 759 | { value: "squash", label: "Squash and merge" }, |
| 760 | 760 | { value: "rebase", label: "Rebase and merge (fast-forward)" }, |
| 761 | 761 | ]} |
modifiedapps/diffinery/src/web/packages.tsx+1−1View file
| @@ -302,7 +302,7 @@ | ||
| 302 | 302 | const { store, pkg, access } = await loadPackage(c, owner, name, "admin"); |
| 303 | 303 | const visibility = (["private", "public"] as const).find((v) => v === f.s("visibility")); |
| 304 | 304 | const repoName = f.s("repository").trim(); |
| 305 | - const repoId = repoName ? (await loadRepo(c, owner.login, repoName, "admin")).repo.id : null; | |
| 305 | + const repoId = repoName ? (await loadRepo(c, owner.login, repoName, "admin", { whileArchived: true })).repo.id : null; | |
| 306 | 306 | await store.updatePackage(pkg.id, { visibility, repoId }); |
| 307 | 307 | await c.get("dir").audit({ actorId: access.actor?.id, actorLogin: access.actor?.login, action: "package.update", target: `${owner.login}/${pkg.name}`, detail: { visibility, repository: repoName || null } }); |
| 308 | 308 | }, |
modifiedapps/diffinery/src/web/repo-settings.tsx+9−3View file
| @@ -15,7 +15,7 @@ | ||
| 15 | 15 | |
| 16 | 16 | const WEBHOOK_EVENTS = ["push", "pull_request", "pull_request_review", "issues", "issue_comment", "workflow_run", "workflow_dispatch"]; |
| 17 | 17 | |
| 18 | -const load = (c: AppContext) => loadRepo(c, prm(c, "owner"), prm(c, "repo"), "admin"); | |
| 18 | +const load = (c: AppContext, opts: { whileArchived?: boolean } = {}) => loadRepo(c, prm(c, "owner"), prm(c, "repo"), "admin", opts); | |
| 19 | 19 | |
| 20 | 20 | function audit(c: AppContext, l: Loaded, action: string, detail?: Record<string, unknown>): void { |
| 21 | 21 | const p = c.get("principal"); |
| @@ -601,7 +601,7 @@ | ||
| 601 | 601 | }); |
| 602 | 602 | |
| 603 | 603 | post(web, "/:owner/:repo/settings/archive", async (c, f) => { |
| 604 | - const l = await load(c); | |
| 604 | + const l = await load(c, { whileArchived: true }); | |
| 605 | 605 | const archived = f.s("archived") === "1"; |
| 606 | 606 | return act( |
| 607 | 607 | c, |
| @@ -615,7 +615,7 @@ | ||
| 615 | 615 | }); |
| 616 | 616 | |
| 617 | 617 | post(web, "/:owner/:repo/settings/delete", async (c, f) => { |
| 618 | - const l = await load(c); | |
| 618 | + const l = await load(c, { whileArchived: true }); | |
| 619 | 619 | requireAccount(c); |
| 620 | 620 | const full = `${l.repo.owner_login}/${l.repo.name}`; |
| 621 | 621 | return act( |
| @@ -673,6 +673,12 @@ | ||
| 673 | 673 | if (!pattern || /\s/.test(pattern)) throw new HttpError(422, "enter a branch name pattern"); |
| 674 | 674 | const original = f.s("original").trim(); |
| 675 | 675 | const rules = rulesFrom(f); |
| 676 | + // The form doesn't edit which app must report each required check; keep those bindings (set through the API). | |
| 677 | + const before = (await l.stub.listProtections()).find((p) => p.pattern === (original || pattern))?.rules.requiredCheckSources; | |
| 678 | + if (before && rules.requiredChecks?.length) { | |
| 679 | + const kept = Object.fromEntries(Object.entries(before).filter(([name]) => rules.requiredChecks!.includes(name))); | |
| 680 | + if (Object.keys(kept).length) rules.requiredCheckSources = kept; | |
| 681 | + } | |
| 676 | 682 | if (original && original !== pattern) await l.stub.setProtection(original, null); |
| 677 | 683 | await l.stub.setProtection(pattern, rules); |
| 678 | 684 | audit(c, l, "repo.protection.set", { pattern, rules }); |
modifiedapps/diffinery/src/web/ui.tsx+1−1View file
| @@ -81,7 +81,7 @@ | ||
| 81 | 81 | export const Status: FC<{ status?: string | null; conclusion?: string | null }> = ({ status, conclusion }) => { |
| 82 | 82 | if (status && status !== "completed") { |
| 83 | 83 | if (status === "in_progress") return <span class="st spin run" title="In progress">{raw(ICONS.running)}</span>; |
| 84 | - return <span class="st warn" title={status === "waiting" ? "Waiting" : "Queued"}>{raw(ICONS.queued)}</span>; | |
| 84 | + return <span class="st warn" title={status === "waiting" ? "Waiting" : status === "pending" ? "Pending" : "Queued"}>{raw(ICONS.queued)}</span>; | |
| 85 | 85 | } |
| 86 | 86 | switch (conclusion) { |
| 87 | 87 | case "success": |
modifiedapps/diffinery/test/e2e/e2e.test.mjs+710−7View file
| @@ -104,19 +104,22 @@ | ||
| 104 | 104 | }); |
| 105 | 105 | } |
| 106 | 106 | |
| 107 | -/** Claim one waiting self-hosted job and run it with the real agent. */ | |
| 108 | -async function runOneJob(owner, repo, { allowFailure = false } = {}) { | |
| 107 | +/** | |
| 108 | + * Claim one waiting self-hosted job and run it with the real agent. | |
| 109 | + * `envFor(claim)` adds environment for the agent (e.g. HOME, DIFFINERY_DISABLE_BUILTINS). | |
| 110 | + */ | |
| 111 | +async function runOneJob(owner, repo, { allowFailure = false, envFor = () => ({}), timeoutMs = 180_000, labels = ["linux"] } = {}) { | |
| 109 | 112 | const claim = await until("a job to claim", async () => { |
| 110 | - const r = await api("POST", `/repos/${owner}/${repo}/actions/runners/claim`, { name: "e2e", labels: ["linux"] }); | |
| 113 | + const r = await api("POST", `/repos/${owner}/${repo}/actions/runners/claim`, { name: "e2e", labels }); | |
| 111 | 114 | return r.status === 200 ? r.json : null; |
| 112 | 115 | }); |
| 113 | - const jobEnv = { DIFFINERY_API_URL: claim.api_url, DIFFINERY_JOB_ID: claim.job_id, DIFFINERY_JOB_TOKEN: claim.token }; | |
| 116 | + const jobEnv = { ...envFor(claim), DIFFINERY_API_URL: claim.api_url, DIFFINERY_JOB_ID: claim.job_id, DIFFINERY_JOB_TOKEN: claim.token }; | |
| 114 | 117 | // E2E_RUNNER_IMAGE runs the job inside the real runner image instead of the local agent bundle. |
| 115 | 118 | const image = process.env.E2E_RUNNER_IMAGE; |
| 116 | 119 | // Spawned asynchronously: a blocked event loop leaves undici holding keep-alive sockets the server already closed. |
| 117 | 120 | const r = image |
| 118 | - ? await spawnAsync("docker", ["run", "--rm", "--network", "host", ...Object.entries(jobEnv).flatMap(([k, v]) => ["-e", `${k}=${v}`]), image], {}, 300_000) | |
| 119 | - : await spawnAsync(process.execPath, [AGENT], { ...process.env, ...jobEnv, DIFFINERY_WORK_ROOT: join(work, `job-${claim.job_id}`) }, 180_000); | |
| 121 | + ? await spawnAsync("docker", ["run", "--rm", "--network", "host", ...Object.entries(jobEnv).flatMap(([k, v]) => ["-e", `${k}=${v}`]), image], {}, timeoutMs + 120_000) | |
| 122 | + : await spawnAsync(process.execPath, [AGENT], { ...process.env, ...jobEnv, DIFFINERY_WORK_ROOT: join(work, `job-${claim.job_id}`) }, timeoutMs); | |
| 120 | 123 | const out = `${r.stdout}${r.stderr}`; |
| 121 | 124 | if (r.status !== 0 && !allowFailure) { |
| 122 | 125 | const log = await fetch(`${base}/api/v3/repos/${owner}/${repo}/actions/jobs/${claim.job_id}/logs`, { headers: { authorization: `Bearer ${token}` } }).then((x) => x.text(), (e) => String(e)); |
| @@ -134,7 +137,7 @@ | ||
| 134 | 137 | execFileSync("pnpm", ["exec", "wrangler", "d1", "migrations", "apply", "diffinery-dev", "--local", "-e", "dev", "--persist-to", persist], { cwd: APP, stdio: "pipe", env: { ...process.env, CI: "1" } }); |
| 135 | 138 | // Until the web UI exists, run the same app through the API-only entry. |
| 136 | 139 | const entry = existsSync(join(APP, "src/web/routes.tsx")) ? [] : ["test/e2e/entry.ts"]; |
| 137 | - dev = spawn("pnpm", ["exec", "wrangler", "dev", ...entry, "-e", "dev", "--ip", "127.0.0.1", "--port", String(port), "--persist-to", persist, "--var", `PUBLIC_URL:${base}`, "--var", `MASTER_KEY:${randomBytes(32).toString("base64")}`, "--show-interactive-dev-session=false"], { | |
| 140 | + dev = spawn("pnpm", ["exec", "wrangler", "dev", ...entry, "-e", "dev", "--ip", "127.0.0.1", "--port", String(port), "--inspector-port", String(await freePort()), "--persist-to", persist, "--var", `PUBLIC_URL:${base}`, "--var", `MASTER_KEY:${randomBytes(32).toString("base64")}`, "--show-interactive-dev-session=false"], { | |
| 138 | 141 | cwd: APP, |
| 139 | 142 | env: { ...process.env, CI: "1", WRANGLER_SEND_METRICS: "false" }, |
| 140 | 143 | stdio: ["ignore", "pipe", "pipe"], |
| @@ -463,6 +466,144 @@ | ||
| 463 | 466 | assert.deepEqual(jobs.filter((j) => j.conclusion === "skipped").map((j) => j.name).sort(), ["nope / say", "nope / shout"]); |
| 464 | 467 | }); |
| 465 | 468 | |
| 469 | +test("job graph: needs-only scheduling, skip cascades, per-leg runs-on, continue-on-error, job concurrency and public payloads", async () => { | |
| 470 | + const owner = (await ok("GET", "/user")).login; | |
| 471 | + await ok("POST", "/user/repos", { name: "graph", private: false }); | |
| 472 | + const src = join(work, "graph"); | |
| 473 | + mkdirSync(join(src, ".github/workflows"), { recursive: true }); | |
| 474 | + writeFileSync( | |
| 475 | + join(src, ".github/workflows/graph.yml"), | |
| 476 | + `name: Graph | |
| 477 | +on: push | |
| 478 | +jobs: | |
| 479 | + slow: | |
| 480 | + runs-on: [self-hosted, e2e-slow] | |
| 481 | + steps: | |
| 482 | + - run: echo slow | |
| 483 | + fast: | |
| 484 | + runs-on: [self-hosted, linux] | |
| 485 | + steps: | |
| 486 | + - run: echo "private=\${{ github.event.repository.private }} visibility=\${{ github.event.repository.visibility }}" | |
| 487 | + after-fast: | |
| 488 | + needs: fast | |
| 489 | + runs-on: [self-hosted, linux] | |
| 490 | + steps: | |
| 491 | + - run: echo after-fast | |
| 492 | + gate: | |
| 493 | + if: \${{ github.event_name == 'nope' }} | |
| 494 | + runs-on: [self-hosted, linux] | |
| 495 | + steps: | |
| 496 | + - run: echo never | |
| 497 | + middle: | |
| 498 | + needs: gate | |
| 499 | + if: \${{ always() }} | |
| 500 | + runs-on: [self-hosted, linux] | |
| 501 | + steps: | |
| 502 | + - run: echo "gate=\${{ needs.gate.result }}" | |
| 503 | + leaf: | |
| 504 | + needs: middle | |
| 505 | + runs-on: [self-hosted, linux] | |
| 506 | + steps: | |
| 507 | + - run: echo leaf | |
| 508 | + rescued: | |
| 509 | + needs: middle | |
| 510 | + if: \${{ !cancelled() && !failure() }} | |
| 511 | + runs-on: [self-hosted, linux] | |
| 512 | + steps: | |
| 513 | + - run: echo "middle=\${{ needs.middle.result }}" | |
| 514 | + lanes: | |
| 515 | + name: lane \${{ matrix.lane }} | |
| 516 | + strategy: | |
| 517 | + matrix: | |
| 518 | + include: | |
| 519 | + - lane: red | |
| 520 | + runner: [self-hosted, e2e-red] | |
| 521 | + - lane: blue | |
| 522 | + runner: [self-hosted, e2e-blue] | |
| 523 | + runs-on: \${{ matrix.runner }} | |
| 524 | + continue-on-error: \${{ matrix.lane == 'red' }} | |
| 525 | + environment: e2e-\${{ matrix.lane }}-\${{ strategy.job-total }} | |
| 526 | + steps: | |
| 527 | + - run: echo "lane=\${{ matrix.lane }}" && test "\${{ matrix.lane }}" = blue | |
| 528 | + after-lanes: | |
| 529 | + needs: lanes | |
| 530 | + runs-on: [self-hosted, linux] | |
| 531 | + steps: | |
| 532 | + - run: echo "lanes=\${{ needs.lanes.result }}" | |
| 533 | + deploy: | |
| 534 | + strategy: | |
| 535 | + matrix: | |
| 536 | + target: [prod, staging] | |
| 537 | + concurrency: e2e-\${{ matrix.target }} | |
| 538 | + runs-on: [self-hosted, e2e-deploy] | |
| 539 | + steps: | |
| 540 | + - run: echo "deploy \${{ matrix.target }}" | |
| 541 | + hotfix: | |
| 542 | + strategy: | |
| 543 | + matrix: | |
| 544 | + target: [prod] | |
| 545 | + concurrency: e2e-\${{ matrix.target }} | |
| 546 | + runs-on: [self-hosted, e2e-deploy] | |
| 547 | + steps: | |
| 548 | + - run: echo "hotfix \${{ matrix.target }}" | |
| 549 | +`, | |
| 550 | + ); | |
| 551 | + git(src, ["init", "-q", "-b", "main"]); | |
| 552 | + git(src, ["add", "."]); | |
| 553 | + git(src, ["commit", "-q", "-m", "graph"]); | |
| 554 | + git(src, ["remote", "add", "origin", `${base}/${owner}/graph.git`]); | |
| 555 | + git(src, ["push", "-q", "origin", "main"]); | |
| 556 | + | |
| 557 | + const run = await until("graph run", async () => (await ok("GET", `/repos/${owner}/graph/actions/runs`)).workflow_runs[0]); | |
| 558 | + const logOf = async (id) => (await fetch(`${base}/api/v3/repos/${owner}/graph/actions/jobs/${id}/logs`, { headers: { authorization: `Bearer ${token}` } })).text(); | |
| 559 | + const jobs = async () => (await ok("GET", `/repos/${owner}/graph/actions/runs/${run.id}/jobs`)).jobs; | |
| 560 | + | |
| 561 | + // Jobs start when their own needs finish: after-fast and rescued run while slow (no runner offers e2e-slow) still waits. | |
| 562 | + const linux = {}; | |
| 563 | + for (let i = 0; i < 4; i++) { | |
| 564 | + const j = await runOneJob(owner, "graph"); | |
| 565 | + linux[j.claim.name] = j.claim.job_id; | |
| 566 | + } | |
| 567 | + assert.deepEqual(Object.keys(linux).sort(), ["after-fast", "fast", "middle", "rescued"]); | |
| 568 | + assert.equal((await jobs()).find((j) => j.name === "slow").status, "waiting"); | |
| 569 | + assert.match(await logOf(linux.fast), /private=false visibility=public/); | |
| 570 | + assert.match(await logOf(linux.middle), /gate=skipped/); | |
| 571 | + assert.match(await logOf(linux.rescued), /middle=success/); | |
| 572 | + | |
| 573 | + // runs-on per matrix leg; the red leg fails but continues on error, so fail-fast spares the blue one. | |
| 574 | + const red = await runOneJob(owner, "graph", { labels: ["e2e-red"], allowFailure: true }); | |
| 575 | + assert.equal(red.claim.name, "lane red"); | |
| 576 | + const blue = await runOneJob(owner, "graph", { labels: ["e2e-blue"] }); | |
| 577 | + assert.equal(blue.claim.name, "lane blue"); | |
| 578 | + const afterLanes = await runOneJob(owner, "graph"); | |
| 579 | + assert.equal(afterLanes.claim.name, "after-lanes"); | |
| 580 | + assert.match(await logOf(afterLanes.claim.job_id), /lanes=success/); | |
| 581 | + const envs = (await ok("GET", `/repos/${owner}/graph/environments`)).environments.map((e) => e.name); | |
| 582 | + assert.ok(envs.includes("e2e-red-2") && envs.includes("e2e-blue-2"), `per-leg environments: ${envs}`); | |
| 583 | + | |
| 584 | + // Job-level concurrency, evaluated per leg: one of the two prod jobs waits as pending until the other finishes. | |
| 585 | + const pending = await until("a pending prod job", async () => (await jobs()).find((j) => j.status === "pending")); | |
| 586 | + assert.ok(["deploy (prod)", "hotfix (prod)"].includes(pending.name), pending.name); | |
| 587 | + const first = await runOneJob(owner, "graph", { labels: ["e2e-deploy"] }); | |
| 588 | + const second = await runOneJob(owner, "graph", { labels: ["e2e-deploy"] }); | |
| 589 | + assert.deepEqual([first.claim.name, second.claim.name].sort(), ["deploy (prod)", "deploy (staging)", "hotfix (prod)"].filter((n) => n !== pending.name).sort()); | |
| 590 | + const third = await runOneJob(owner, "graph", { labels: ["e2e-deploy"] }); | |
| 591 | + assert.equal(third.claim.name, pending.name); | |
| 592 | + | |
| 593 | + const slow = await runOneJob(owner, "graph", { labels: ["e2e-slow"] }); | |
| 594 | + assert.equal(slow.claim.name, "slow"); | |
| 595 | + const done = await until("graph run to complete", async () => { | |
| 596 | + const r = await ok("GET", `/repos/${owner}/graph/actions/runs/${run.id}`); | |
| 597 | + return r.status === "completed" ? r : null; | |
| 598 | + }); | |
| 599 | + assert.equal(done.conclusion, "success", "a failed leg with continue-on-error does not fail the run"); | |
| 600 | + const byName = Object.fromEntries((await jobs()).map((j) => [j.name, j])); | |
| 601 | + assert.equal(byName.gate.conclusion, "skipped"); | |
| 602 | + assert.equal(byName.leaf.conclusion, "skipped", "a skipped ancestor skips the jobs below it"); | |
| 603 | + assert.equal(byName["lane red"].conclusion, "failure"); | |
| 604 | + assert.equal(byName["lane blue"].conclusion, "success"); | |
| 605 | +}); | |
| 606 | + | |
| 466 | 607 | test("workflow_run: a workflow follows another one on the default branch", async () => { |
| 467 | 608 | const owner = (await ok("GET", "/user")).login; |
| 468 | 609 | await ok("POST", "/user/repos", { name: "chain" }); |
| @@ -689,6 +830,298 @@ | ||
| 689 | 830 | assert.match(await page(`/${owner}/queue/settings/branches`), /merge queue/); |
| 690 | 831 | }); |
| 691 | 832 | |
| 833 | +const POLICY_WORKFLOW = `name: Policy | |
| 834 | +on: | |
| 835 | + push: | |
| 836 | + branches: [main, agent-work] | |
| 837 | + pull_request: | |
| 838 | +jobs: | |
| 839 | + probe: | |
| 840 | + runs-on: [self-hosted, linux] | |
| 841 | + permissions: { contents: write, pull-requests: write } | |
| 842 | + steps: | |
| 843 | + - uses: actions/checkout@v4 | |
| 844 | + with: { ref: main } | |
| 845 | + - name: Push to main, push a workflow change, approve the pull request | |
| 846 | + env: | |
| 847 | + GH_TOKEN: \${{ github.token }} | |
| 848 | + PR: \${{ github.event.pull_request.number }} | |
| 849 | + GIT_AUTHOR_NAME: bot | |
| 850 | + GIT_AUTHOR_EMAIL: bot@example.com | |
| 851 | + GIT_COMMITTER_NAME: bot | |
| 852 | + GIT_COMMITTER_EMAIL: bot@example.com | |
| 853 | + run: | | |
| 854 | + echo "run \${{ github.run_id }}" > note.txt | |
| 855 | + git add note.txt && git commit -qm note | |
| 856 | + if git push -q origin HEAD:main 2>err.txt; then echo "main push: ok"; else echo "main push: refused $(tr '\\n' ' ' < err.txt)"; fi | |
| 857 | + git reset -q --hard HEAD~1 | |
| 858 | + echo "# edited" >> .github/workflows/policy.yml | |
| 859 | + git commit -qam "edit the workflow" | |
| 860 | + if git push -q origin "HEAD:refs/heads/wf-\${{ github.run_id }}" 2>err.txt; then echo "workflow push: ok"; else echo "workflow push: refused $(tr '\\n' ' ' < err.txt)"; fi | |
| 861 | + if [ -n "$PR" ]; then | |
| 862 | + code=$(curl -s -o out.json -w '%{http_code}' -X POST -H "authorization: Bearer $GH_TOKEN" -H 'content-type: application/json' -d '{"event":"APPROVE","body":"from a workflow"}' "$GITHUB_API_URL/repos/$GITHUB_REPOSITORY/pulls/$PR/reviews") | |
| 863 | + echo "approve: $code $(cat out.json)" | |
| 864 | + fi | |
| 865 | + defaults: | |
| 866 | + if: github.event_name == 'push' && github.ref == 'refs/heads/main' | |
| 867 | + runs-on: [self-hosted, linux] | |
| 868 | + steps: | |
| 869 | + - env: { GH_TOKEN: "\${{ github.token }}" } | |
| 870 | + run: | | |
| 871 | + code=$(curl -s -o out.json -w '%{http_code}' -X POST -H "authorization: Bearer $GH_TOKEN" -H 'content-type: application/json' -d '{"title":"from the default token"}' "$GITHUB_API_URL/repos/$GITHUB_REPOSITORY/issues") | |
| 872 | + echo "default token issue write: $code $(cat out.json)" | |
| 873 | + code=$(curl -s -o /dev/null -w '%{http_code}' -H "authorization: Bearer $GH_TOKEN" "$GITHUB_API_URL/repos/$GITHUB_REPOSITORY/contents/README.md") | |
| 874 | + echo "default token contents read: $code" | |
| 875 | +`; | |
| 876 | + | |
| 877 | +test("workflow tokens: read-only default, no approvals or workflow edits, agent code stays an agent", { timeout: 600_000 }, async () => { | |
| 878 | + const owner = (await ok("GET", "/user")).login; | |
| 879 | + await ok("POST", "/user/repos", { name: "policy" }); | |
| 880 | + const settingsPath = `/repos/${owner}/policy/actions/permissions/workflow`; | |
| 881 | + // New repositories start like GitHub's: read-only token, no pull request approvals. | |
| 882 | + assert.deepEqual(await ok("GET", settingsPath), { default_workflow_permissions: "read", can_approve_pull_request_reviews: false }); | |
| 883 | + const bot = (await ok("POST", "/diffinery/agents", { login: "policy-bot" })).token.token; | |
| 884 | + assert.equal((await api("PUT", settingsPath, { can_approve_pull_request_reviews: true }, bot)).status, 403, "agents cannot change what workflows may do"); | |
| 885 | + assert.equal((await api("PUT", settingsPath, { default_workflow_permissions: "admin" })).status, 422); | |
| 886 | + | |
| 887 | + const src = join(work, "policy"); | |
| 888 | + mkdirSync(join(src, ".github/workflows"), { recursive: true }); | |
| 889 | + writeFileSync(join(src, "README.md"), "# policy\n"); | |
| 890 | + writeFileSync(join(src, ".github/workflows/policy.yml"), POLICY_WORKFLOW); | |
| 891 | + git(src, ["init", "-q", "-b", "main"]); | |
| 892 | + git(src, ["add", "-A"]); | |
| 893 | + git(src, ["commit", "-q", "-m", "policy"]); | |
| 894 | + git(src, ["remote", "add", "origin", `${base}/${owner}/policy.git`]); | |
| 895 | + git(src, ["push", "-q", "origin", "main"]); | |
| 896 | + | |
| 897 | + /** Runs `count` jobs; returns their logs by job name. */ | |
| 898 | + const runJobs = async (count) => { | |
| 899 | + const logs = {}; | |
| 900 | + for (let i = 0; i < count; i++) { | |
| 901 | + const { claim } = await runOneJob(owner, "policy"); | |
| 902 | + const raw = await (await fetch(`${base}/api/v3/repos/${owner}/policy/actions/jobs/${claim.job_id}/logs`, { headers: { authorization: `Bearer ${token}` } })).text(); | |
| 903 | + logs[claim.name] = raw.replace(/^\S+Z /gm, ""); | |
| 904 | + } | |
| 905 | + return logs; | |
| 906 | + }; | |
| 907 | + const refused = (path) => new RegExp(`workflow push: refused .*refusing to allow a GitHub App to create or update workflow \`${path.replace(/\./g, "\\.")}\` without \`workflows\` permission`); | |
| 908 | + | |
| 909 | + // A person's push: the job token writes what it was granted, but never workflow files; | |
| 910 | + // the default token (no permissions:) only reads. | |
| 911 | + let logs = await runJobs(2); | |
| 912 | + assert.match(logs.probe, /main push: ok/); | |
| 913 | + assert.match(logs.probe, refused(".github/workflows/policy.yml")); | |
| 914 | + assert.match(logs.defaults, /default token issue write: 403 .*issues: write/); | |
| 915 | + assert.match(logs.defaults, /default token contents read: 200/); | |
| 916 | + const branches = (await ok("GET", `/repos/${owner}/policy/branches`)).map((b) => b.name); | |
| 917 | + assert.ok(!branches.some((b) => b.startsWith("wf-")), JSON.stringify(branches)); | |
| 918 | + git(src, ["pull", "-q", "--ff-only", "origin", "main"]); | |
| 919 | + assert.match(readFileSync(join(src, "note.txt"), "utf8"), /^run \d+/); | |
| 920 | + | |
| 921 | + // An agent's push: the run executes the agent's code, so its token is an agent's and may not push to the default branch. | |
| 922 | + const agentClone = join(work, "policy-agent"); | |
| 923 | + git(work, ["clone", "-q", `${base}/${owner}/policy.git`, agentClone], bot); | |
| 924 | + git(agentClone, ["checkout", "-q", "-b", "agent-work"], bot); | |
| 925 | + writeFileSync(join(agentClone, "agent.txt"), "from the agent\n"); | |
| 926 | + git(agentClone, ["add", "agent.txt"], bot); | |
| 927 | + git(agentClone, ["commit", "-q", "-m", "agent change"], bot); | |
| 928 | + git(agentClone, ["push", "-q", "origin", "agent-work"], bot); | |
| 929 | + logs = await runJobs(1); | |
| 930 | + assert.match(logs.probe, /main push: refused .*agents may not push to protected or default branches/); | |
| 931 | + assert.match(logs.probe, refused(".github/workflows/policy.yml")); | |
| 932 | + | |
| 933 | + // A person opens a pull request from the branch the agent pushed: still the agent's code. | |
| 934 | + // Workflows may not approve pull requests unless the repository allows it. | |
| 935 | + const pr1 = await ok("POST", `/repos/${owner}/policy/pulls`, { title: "Agent work", head: "agent-work", base: "main" }); | |
| 936 | + logs = await runJobs(1); | |
| 937 | + assert.match(logs.probe, /main push: refused .*agents may not push/); | |
| 938 | + assert.match(logs.probe, /approve: 403 .*GitHub Actions is not permitted to approve pull requests/); | |
| 939 | + assert.equal((await ok("GET", `/repos/${owner}/policy/pulls/${pr1.number}/reviews`)).length, 0); | |
| 940 | + | |
| 941 | + // Allowed: the approval is recorded, but only people's approvals count toward the required ones. | |
| 942 | + await ok("PUT", settingsPath, { can_approve_pull_request_reviews: true }); | |
| 943 | + assert.deepEqual(await ok("GET", settingsPath), { default_workflow_permissions: "read", can_approve_pull_request_reviews: true }); | |
| 944 | + await ok("PUT", `/repos/${owner}/policy/branches/main/protection`, { | |
| 945 | + requirePullRequest: true, | |
| 946 | + requiredApprovals: 1, | |
| 947 | + dismissStaleReviews: false, | |
| 948 | + requiredChecks: [], | |
| 949 | + allowForcePush: false, | |
| 950 | + allowDeletion: false, | |
| 951 | + requireLinearHistory: false, | |
| 952 | + enforceAdmins: true, | |
| 953 | + agentsMayPush: false, | |
| 954 | + agentsMayMerge: false, | |
| 955 | + }); | |
| 956 | + git(src, ["checkout", "-q", "-b", "feature"]); | |
| 957 | + writeFileSync(join(src, "feature.txt"), "a person's change\n"); | |
| 958 | + git(src, ["add", "feature.txt"]); | |
| 959 | + git(src, ["commit", "-q", "-m", "feature"]); | |
| 960 | + git(src, ["push", "-q", "origin", "feature"]); | |
| 961 | + const pr2 = await ok("POST", `/repos/${owner}/policy/pulls`, { title: "Feature", head: "feature", base: "main" }); | |
| 962 | + logs = await runJobs(1); | |
| 963 | + assert.match(logs.probe, /main push: refused .*pull request/); | |
| 964 | + assert.match(logs.probe, /approve: 20[01] /); | |
| 965 | + const reviews = await ok("GET", `/repos/${owner}/policy/pulls/${pr2.number}/reviews`); | |
| 966 | + assert.deepEqual(reviews.map((r) => r.state), ["APPROVED"]); | |
| 967 | + const m = await ok("GET", `/repos/${owner}/policy/pulls/${pr2.number}/merge`); | |
| 968 | + assert.equal(m.mergeable, false, JSON.stringify(m)); | |
| 969 | + assert.equal(m.approvals, 0, JSON.stringify(m)); | |
| 970 | + assert.match(JSON.stringify(m.blockers), /needs 1 more approving review/); | |
| 971 | +}); | |
| 972 | + | |
| 973 | +const SCOPES_WORKFLOW = `name: Scopes | |
| 974 | +on: | |
| 975 | + push: | |
| 976 | + branches: [main, a, b] | |
| 977 | + pull_request: | |
| 978 | +jobs: | |
| 979 | + c: | |
| 980 | + runs-on: [self-hosted, linux] | |
| 981 | + steps: | |
| 982 | + - uses: actions/cache@v4 | |
| 983 | + with: { path: built, key: "b-\${{ github.run_id }}", restore-keys: "b-" } | |
| 984 | + - uses: actions/cache@v4 | |
| 985 | + env: { DIFFINERY_BUILTINS: "false" } | |
| 986 | + with: { path: real, key: "r-\${{ github.run_id }}", restore-keys: "r-" } | |
| 987 | + - run: | | |
| 988 | + echo "restored built=[$(cat built/from 2>/dev/null)] real=[$(cat real/from 2>/dev/null)]" | |
| 989 | + mkdir -p built real && echo "$GITHUB_REF" > built/from && echo "$GITHUB_REF" > real/from | |
| 990 | + - if: github.ref == 'refs/heads/main' | |
| 991 | + uses: actions/upload-artifact@v4 | |
| 992 | + with: { name: "builtin-\${{ github.run_id }}", path: built/from, retention-days: 1 } | |
| 993 | + - if: github.ref == 'refs/heads/main' | |
| 994 | + uses: actions/upload-artifact@v4 | |
| 995 | + env: { DIFFINERY_BUILTINS: "false" } | |
| 996 | + with: { name: "real-\${{ github.run_id }}", path: built/from, retention-days: 3 } | |
| 997 | + - if: github.ref == 'refs/heads/main' | |
| 998 | + uses: actions/upload-artifact@v4 | |
| 999 | + with: { name: "capped-\${{ github.run_id }}", path: built/from, retention-days: 400 } | |
| 1000 | +`; | |
| 1001 | + | |
| 1002 | +test("caches are scoped to branches and pull requests; artifacts keep their retention-days", { timeout: 900_000 }, async () => { | |
| 1003 | + const owner = (await ok("GET", "/user")).login; | |
| 1004 | + await ok("POST", "/user/repos", { name: "scopes" }); | |
| 1005 | + const src = join(work, "scopes"); | |
| 1006 | + mkdirSync(join(src, ".github/workflows"), { recursive: true }); | |
| 1007 | + writeFileSync(join(src, ".github/workflows/scopes.yml"), SCOPES_WORKFLOW); | |
| 1008 | + writeFileSync(join(src, "README.md"), "# scopes\n"); | |
| 1009 | + git(src, ["init", "-q", "-b", "main"]); | |
| 1010 | + git(src, ["add", "-A"]); | |
| 1011 | + git(src, ["commit", "-q", "-m", "scopes"]); | |
| 1012 | + git(src, ["remote", "add", "origin", `${base}/${owner}/scopes.git`]); | |
| 1013 | + /** Runs the job a push or pull request started; returns what its caches restored. */ | |
| 1014 | + const restored = async () => { | |
| 1015 | + const { claim } = await runOneJob(owner, "scopes", { timeoutMs: 300_000 }); | |
| 1016 | + const log = await (await fetch(`${base}/api/v3/repos/${owner}/scopes/actions/jobs/${claim.job_id}/logs`, { headers: { authorization: `Bearer ${token}` } })).text(); | |
| 1017 | + const m = /^\S+ restored built=\[(.*?)\] real=\[(.*?)\]$/m.exec(log); | |
| 1018 | + assert.ok(m, log); | |
| 1019 | + return { builtin: m[1], real: m[2] }; | |
| 1020 | + }; | |
| 1021 | + const branch = (name, from, file) => { | |
| 1022 | + git(src, ["checkout", "-q", "-b", name, from]); | |
| 1023 | + if (file) { | |
| 1024 | + writeFileSync(join(src, file), `${name}\n`); | |
| 1025 | + git(src, ["add", file]); | |
| 1026 | + git(src, ["commit", "-q", "-m", name]); | |
| 1027 | + } | |
| 1028 | + git(src, ["push", "-q", "origin", name]); | |
| 1029 | + }; | |
| 1030 | + | |
| 1031 | + // The default branch starts empty and saves its own entries. | |
| 1032 | + git(src, ["push", "-q", "origin", "main"]); | |
| 1033 | + assert.deepEqual(await restored(), { builtin: "", real: "" }); | |
| 1034 | + // A branch falls back to the default branch's caches... | |
| 1035 | + branch("a", "main", "a.txt"); | |
| 1036 | + assert.deepEqual(await restored(), { builtin: "refs/heads/main", real: "refs/heads/main" }); | |
| 1037 | + // ...but never sees another branch's, though a's are newer and match the restore key. | |
| 1038 | + branch("b", "main", "b.txt"); | |
| 1039 | + assert.deepEqual(await restored(), { builtin: "refs/heads/main", real: "refs/heads/main" }); | |
| 1040 | + // A pull request reads its base branch's caches. | |
| 1041 | + branch("c", "a", "c.txt"); | |
| 1042 | + await ok("POST", `/repos/${owner}/scopes/pulls`, { title: "c into a", head: "c", base: "a" }); | |
| 1043 | + assert.deepEqual(await restored(), { builtin: "refs/heads/a", real: "refs/heads/a" }); | |
| 1044 | + // Another pull request doesn't see the first one's (saved under refs/pull/<n>/merge). | |
| 1045 | + branch("d", "main", "d.txt"); | |
| 1046 | + await ok("POST", `/repos/${owner}/scopes/pulls`, { title: "d into main", head: "d", base: "main" }); | |
| 1047 | + assert.deepEqual(await restored(), { builtin: "refs/heads/main", real: "refs/heads/main" }); | |
| 1048 | + const caches = await ok("GET", `/repos/${owner}/scopes/actions/caches?per_page=100`); | |
| 1049 | + assert.deepEqual([...new Set(caches.actions_caches.map((c) => c.ref))].sort(), ["refs/heads/a", "refs/heads/b", "refs/heads/main", "refs/pull/1/merge", "refs/pull/2/merge"]); | |
| 1050 | + | |
| 1051 | + // retention-days (the built-in and the real upload-artifact), capped at 90 days. | |
| 1052 | + const days = (iso) => Math.round((Date.parse(iso) - Date.now()) / 86_400_000); | |
| 1053 | + const artifacts = (await ok("GET", `/repos/${owner}/scopes/actions/runs/1/artifacts`)).artifacts; | |
| 1054 | + const byName = Object.fromEntries(artifacts.map((a) => [a.name.replace(/-\d+$/, ""), a])); | |
| 1055 | + assert.equal(days(byName.builtin.expires_at), 1, JSON.stringify(byName.builtin)); | |
| 1056 | + assert.equal(days(byName.real.expires_at), 3, JSON.stringify(byName.real)); | |
| 1057 | + assert.equal(days(byName.capped.expires_at), 90, JSON.stringify(byName.capped)); | |
| 1058 | + assert.ok(artifacts.every((a) => a.expired === false)); | |
| 1059 | +}); | |
| 1060 | + | |
| 1061 | +test("an older runner's cache API is scoped the same way", async () => { | |
| 1062 | + const owner = (await ok("GET", "/user")).login; | |
| 1063 | + await ok("POST", "/user/repos", { name: "legacy-cache" }); | |
| 1064 | + const src = join(work, "legacy-cache"); | |
| 1065 | + mkdirSync(join(src, ".github/workflows"), { recursive: true }); | |
| 1066 | + writeFileSync(join(src, ".github/workflows/w.yml"), "on:\n push:\n branches: [main, a, b]\njobs:\n j:\n runs-on: [self-hosted, linux]\n steps: [{ run: echo }]\n"); | |
| 1067 | + git(src, ["init", "-q", "-b", "main"]); | |
| 1068 | + git(src, ["add", "-A"]); | |
| 1069 | + git(src, ["commit", "-q", "-m", "legacy"]); | |
| 1070 | + git(src, ["remote", "add", "origin", `${base}/${owner}/legacy-cache.git`]); | |
| 1071 | + | |
| 1072 | + /** Claims the job a push started and starts it, as an older runner would, without running steps. */ | |
| 1073 | + const startJob = async () => { | |
| 1074 | + const claim = await until("a job to claim", async () => { | |
| 1075 | + const r = await api("POST", `/repos/${owner}/legacy-cache/actions/runners/claim`, { name: "old-runner", labels: ["linux"] }); | |
| 1076 | + return r.status === 200 ? r.json : null; | |
| 1077 | + }); | |
| 1078 | + const h = { authorization: `Bearer ${claim.token}` }; | |
| 1079 | + const at = `${base}/runner/jobs/${claim.job_id}`; | |
| 1080 | + assert.equal((await fetch(at, { headers: h })).status, 200); | |
| 1081 | + return { | |
| 1082 | + put: async (key, body) => (await fetch(`${at}/cache/${encodeURIComponent(key)}`, { method: "PUT", headers: { ...h, "content-length": String(body.length) }, body })).status, | |
| 1083 | + lookup: async (key, restore = "") => { | |
| 1084 | + const r = await fetch(`${at}/cache?key=${encodeURIComponent(key)}&restore=${encodeURIComponent(restore)}`, { headers: h }); | |
| 1085 | + if (r.status !== 200) return { status: r.status }; | |
| 1086 | + const hit = await r.json(); | |
| 1087 | + const blob = await fetch(hit.url, { headers: h }); | |
| 1088 | + return { key: hit.key, content: blob.status === 200 ? await blob.text() : blob.status }; | |
| 1089 | + }, | |
| 1090 | + blob: async (stored) => (await fetch(`${at}/cache/blob/${encodeURIComponent(stored)}`, { headers: h })).status, | |
| 1091 | + done: async () => assert.equal((await fetch(`${at}/complete`, { method: "POST", headers: { ...h, "content-type": "application/json" }, body: JSON.stringify({ conclusion: "success", outputs: {} }) })).status, 200), | |
| 1092 | + }; | |
| 1093 | + }; | |
| 1094 | + | |
| 1095 | + const branch = (name) => { | |
| 1096 | + git(src, ["checkout", "-q", "-b", name, "main"]); | |
| 1097 | + writeFileSync(join(src, `${name}.txt`), `${name}\n`); | |
| 1098 | + git(src, ["add", "-A"]); | |
| 1099 | + git(src, ["commit", "-q", "-m", name]); | |
| 1100 | + git(src, ["push", "-q", "origin", name]); | |
| 1101 | + }; | |
| 1102 | + | |
| 1103 | + git(src, ["push", "-q", "origin", "main"]); | |
| 1104 | + let job = await startJob(); | |
| 1105 | + assert.equal(await job.put("lk-main", "from main"), 200); | |
| 1106 | + await job.done(); | |
| 1107 | + | |
| 1108 | + branch("a"); | |
| 1109 | + job = await startJob(); | |
| 1110 | + assert.deepEqual(await job.lookup("lk-a", "lk-"), { key: "lk-main", content: "from main" }); | |
| 1111 | + assert.equal(await job.put("lk-a", "from a"), 200); | |
| 1112 | + assert.deepEqual(await job.lookup("lk-a"), { key: "lk-a", content: "from a" }); | |
| 1113 | + await job.done(); | |
| 1114 | + | |
| 1115 | + branch("b"); | |
| 1116 | + job = await startJob(); | |
| 1117 | + assert.deepEqual(await job.lookup("lk-a"), { status: 404 }); | |
| 1118 | + assert.deepEqual(await job.lookup("lk-x", "lk-"), { key: "lk-main", content: "from main" }); | |
| 1119 | + // a's entry can't be fetched directly either | |
| 1120 | + assert.equal(await job.blob("refs/heads/a\nlk-a"), 404); | |
| 1121 | + assert.equal(await job.blob("lk-a"), 404); | |
| 1122 | + await job.done(); | |
| 1123 | +}); | |
| 1124 | + | |
| 692 | 1125 | const hasDocker = spawnSync("docker", ["info"], { stdio: "ignore", timeout: 15_000 }).status === 0; |
| 693 | 1126 | |
| 694 | 1127 | test("containers: a job container and service containers on a self-hosted runner", { skip: !hasDocker && "Docker is not available" }, async () => { |
| @@ -737,6 +1170,276 @@ | ||
| 737 | 1170 | assert.equal(done.conclusion, "success"); |
| 738 | 1171 | }); |
| 739 | 1172 | |
| 1173 | +// ---------------------------------------------------------------- the official actions, unmodified | |
| 1174 | + | |
| 1175 | +// No workflow-level permissions: jobs get the repository default (write), like GitHub's permissive default. | |
| 1176 | +const TOOLKIT_WORKFLOW = `name: Toolkit | |
| 1177 | +on: push | |
| 1178 | +jobs: | |
| 1179 | + produce: | |
| 1180 | + runs-on: [self-hosted, linux] | |
| 1181 | + steps: | |
| 1182 | + - uses: actions/checkout@v4 | |
| 1183 | + with: { lfs: true, submodules: true, persist-credentials: false } | |
| 1184 | + - name: LFS, submodules, no persisted credentials | |
| 1185 | + run: | | |
| 1186 | + cat big.bin lib/lib.txt | |
| 1187 | + git config --get-regexp 'extraheader' || echo "no persisted credentials" | |
| 1188 | + - uses: actions/checkout@v4 | |
| 1189 | + with: { path: sparse, sparse-checkout: docs } | |
| 1190 | + - run: echo "sparse=$(ls sparse | tr '\\n' ' ')" | |
| 1191 | + - id: real-cache | |
| 1192 | + uses: actions/cache@v4 | |
| 1193 | + env: { DIFFINERY_BUILTINS: "false" } | |
| 1194 | + with: { path: deps, key: "deps-\${{ hashFiles('lock.txt') }}" } | |
| 1195 | + - if: steps.real-cache.outputs.cache-hit != 'true' | |
| 1196 | + run: mkdir -p deps && echo "saved by run \${{ github.run_id }}" > deps/marker | |
| 1197 | + - run: echo "real-hit=\${{ steps.real-cache.outputs.cache-hit }} $(cat deps/marker)" | |
| 1198 | + - id: builtin-cache | |
| 1199 | + uses: actions/cache/restore@v4 | |
| 1200 | + with: { path: deps, key: "deps-\${{ hashFiles('lock.txt') }}", lookup-only: true } | |
| 1201 | + - id: v3-cache | |
| 1202 | + uses: actions/cache/restore@v3 | |
| 1203 | + env: { DIFFINERY_BUILTINS: "false" } | |
| 1204 | + with: { path: deps, key: "deps-\${{ hashFiles('lock.txt') }}", lookup-only: true } | |
| 1205 | + - run: echo "builtin-hit=\${{ steps.builtin-cache.outputs.cache-hit }} v3-hit=\${{ steps.v3-cache.outputs.cache-hit }}" | |
| 1206 | + - run: mkdir -p dist/sub && echo one > dist/one.txt && echo two > dist/sub/two.txt && echo hidden > dist/.hidden | |
| 1207 | + - id: up-real | |
| 1208 | + uses: actions/upload-artifact@v4 | |
| 1209 | + env: { DIFFINERY_BUILTINS: "false" } | |
| 1210 | + with: { name: dist-real, path: dist/, include-hidden-files: true, retention-days: 3 } | |
| 1211 | + - id: up-builtin | |
| 1212 | + uses: actions/upload-artifact@v4 | |
| 1213 | + with: { name: dist-builtin, path: dist/, compression-level: 0 } | |
| 1214 | + - run: echo "ids=\${{ steps.up-real.outputs.artifact-id }},\${{ steps.up-builtin.outputs.artifact-id }} url=\${{ steps.up-real.outputs.artifact-url }}" | |
| 1215 | + consume: | |
| 1216 | + needs: produce | |
| 1217 | + runs-on: [self-hosted, linux] | |
| 1218 | + steps: | |
| 1219 | + - uses: actions/download-artifact@v4 | |
| 1220 | + env: { DIFFINERY_BUILTINS: "false" } | |
| 1221 | + with: { path: all } | |
| 1222 | + - uses: actions/download-artifact@v4 | |
| 1223 | + with: { pattern: dist-*, path: merged, merge-multiple: true } | |
| 1224 | + - uses: actions/download-artifact@v4 | |
| 1225 | + env: { DIFFINERY_BUILTINS: "false" } | |
| 1226 | + with: { name: dist-builtin, path: one } | |
| 1227 | + - run: find all merged one -type f | LC_ALL=C sort | tr '\\n' ' ' | |
| 1228 | + - uses: actions/checkout@v4 | |
| 1229 | + with: { path: src } | |
| 1230 | + - id: prev | |
| 1231 | + run: echo "id=$(cat src/prev-run.txt 2>/dev/null)" >> "$GITHUB_OUTPUT" | |
| 1232 | + - if: steps.prev.outputs.id != '' | |
| 1233 | + uses: actions/download-artifact@v4 | |
| 1234 | + env: { DIFFINERY_BUILTINS: "false" } | |
| 1235 | + with: { name: dist-real, path: prev-real, run-id: "\${{ steps.prev.outputs.id }}", github-token: "\${{ github.token }}", repository: "\${{ github.repository }}" } | |
| 1236 | + - if: steps.prev.outputs.id != '' | |
| 1237 | + uses: actions/download-artifact@v4 | |
| 1238 | + with: { pattern: "dist-*", path: prev-builtin, run-id: "\${{ steps.prev.outputs.id }}", github-token: "\${{ github.token }}" } | |
| 1239 | + - if: steps.prev.outputs.id != '' | |
| 1240 | + run: echo "previous run $(cat prev-real/sub/two.txt) $(find prev-builtin -type f | LC_ALL=C sort | tr '\\n' ' ')" | |
| 1241 | + script: | |
| 1242 | + runs-on: [self-hosted, linux] | |
| 1243 | + steps: | |
| 1244 | + - id: gs | |
| 1245 | + uses: actions/github-script@v7 | |
| 1246 | + with: | |
| 1247 | + result-encoding: string | |
| 1248 | + script: | | |
| 1249 | + const { owner, repo } = context.repo; | |
| 1250 | + const label = \`from-run-\${context.runId}\`; | |
| 1251 | + await github.rest.issues.createLabel({ owner, repo, name: label, color: "ff0000" }); | |
| 1252 | + const { data: issue } = await github.rest.issues.create({ owner, repo, title: \`Opened by run \${context.runId}\`, body: "hello", labels: [label] }); | |
| 1253 | + await github.rest.issues.createComment({ owner, repo, issue_number: issue.number, body: "commented by github-script" }); | |
| 1254 | + const { data: comments } = await github.rest.issues.listComments({ owner, repo, issue_number: issue.number }); | |
| 1255 | + const { data: got } = await github.rest.issues.get({ owner, repo, issue_number: issue.number }); | |
| 1256 | + const { data: pulls } = await github.rest.pulls.list({ owner, repo, state: "all" }); | |
| 1257 | + await github.rest.repos.createCommitStatus({ owner, repo, sha: context.sha, state: "success", context: "github-script", description: "set with the default GITHUB_TOKEN" }); | |
| 1258 | + return \`issue=\${issue.number} labels=\${got.labels.map((l) => l.name)} comments=\${comments.length} pulls=\${pulls.length} event=\${context.eventName} repo=\${context.payload.repository.full_name} sender=\${context.payload.sender.login}\`; | |
| 1259 | + - run: echo "script says $RESULT" | |
| 1260 | + env: { RESULT: "\${{ steps.gs.outputs.result }}" } | |
| 1261 | + restricted: | |
| 1262 | + runs-on: [self-hosted, linux] | |
| 1263 | + permissions: { contents: read } | |
| 1264 | + steps: | |
| 1265 | + - id: gs | |
| 1266 | + uses: actions/github-script@v7 | |
| 1267 | + with: | |
| 1268 | + result-encoding: string | |
| 1269 | + script: | | |
| 1270 | + const { owner, repo } = context.repo; | |
| 1271 | + const { data: meta } = await github.rest.repos.get({ owner, repo }); | |
| 1272 | + try { | |
| 1273 | + await github.rest.issues.create({ owner, repo, title: "must not be created" }); | |
| 1274 | + return "issue write allowed"; | |
| 1275 | + } catch (e) { | |
| 1276 | + return \`repo=\${meta.name} issue write -> \${e.status}: \${e.message}\`; | |
| 1277 | + } | |
| 1278 | + - run: echo "restricted says $RESULT" | |
| 1279 | + env: { RESULT: "\${{ steps.gs.outputs.result }}" } | |
| 1280 | + node: | |
| 1281 | + runs-on: [self-hosted, linux] | |
| 1282 | + steps: | |
| 1283 | + - uses: actions/checkout@v4 | |
| 1284 | + - id: node | |
| 1285 | + uses: actions/setup-node@v4 | |
| 1286 | + with: { node-version: 20, cache: npm } | |
| 1287 | + - run: npm ci --no-audit --no-fund && echo "node=$(node --version) npm-cache-hit=\${{ steps.node.outputs.cache-hit }}" | |
| 1288 | + java: | |
| 1289 | + runs-on: [self-hosted, linux] | |
| 1290 | + steps: | |
| 1291 | + - uses: actions/checkout@v4 | |
| 1292 | + - id: java | |
| 1293 | + uses: actions/setup-java@v4 | |
| 1294 | + with: { distribution: oracle, java-version: "21", cache: maven } | |
| 1295 | + - run: | | |
| 1296 | + mkdir -p ~/.m2/repository/e2e && echo "run \${{ github.run_id }}" > ~/.m2/repository/e2e/marker | |
| 1297 | + echo "java=$(java -version 2>&1 | grep version) maven-cache-hit=\${{ steps.java.outputs.cache-hit }} $(cat ~/.m2/repository/e2e/marker)" | |
| 1298 | +`; | |
| 1299 | + | |
| 1300 | +/** Runs every job of `run` with the real agent; returns their logs by job name. */ | |
| 1301 | +async function runToolkit(owner, run) { | |
| 1302 | + const home = join(work, "toolkit-home"); | |
| 1303 | + const logs = {}; | |
| 1304 | + for (let i = 0; i < 6; i++) { | |
| 1305 | + const job = await runOneJob(owner, "toolkit", { | |
| 1306 | + timeoutMs: 420_000, | |
| 1307 | + envFor: (claim) => { | |
| 1308 | + // A fresh HOME per job (at the same path, as on hosted runners): caches must come from the cache service. | |
| 1309 | + rmSync(home, { recursive: true, force: true }); | |
| 1310 | + mkdirSync(home, { recursive: true }); | |
| 1311 | + return { | |
| 1312 | + ...(process.env.E2E_RUNNER_IMAGE ? {} : { HOME: home, RUNNER_TOOL_CACHE: join(work, "toolcache") }), | |
| 1313 | + // The node job runs every action for real, checkout and setup-node included. | |
| 1314 | + ...(claim.name === "node" ? { DIFFINERY_DISABLE_BUILTINS: "1" } : {}), | |
| 1315 | + }; | |
| 1316 | + }, | |
| 1317 | + }); | |
| 1318 | + // Without the timestamps, so assertions can span lines. | |
| 1319 | + const raw = await (await fetch(`${base}/api/v3/repos/${owner}/toolkit/actions/jobs/${job.claim.job_id}/logs`, { headers: { authorization: `Bearer ${token}` } })).text(); | |
| 1320 | + logs[job.claim.name] = raw.replace(/^\S+Z /gm, ""); | |
| 1321 | + // Kept with E2E_KEEP=1 for debugging. | |
| 1322 | + mkdirSync(join(work, "toolkit-logs"), { recursive: true }); | |
| 1323 | + writeFileSync(join(work, "toolkit-logs", `run${run.run_number}-${job.claim.name}.log`), `${logs[job.claim.name]}\n--- agent ---\n${job.out}`); | |
| 1324 | + } | |
| 1325 | + const done = await until("toolkit run to complete", async () => { | |
| 1326 | + const r = await ok("GET", `/repos/${owner}/toolkit/actions/runs/${run.id}`); | |
| 1327 | + return r.status === "completed" ? r : null; | |
| 1328 | + }); | |
| 1329 | + assert.equal(done.conclusion, "success", JSON.stringify(logs, null, 2).slice(-8000)); | |
| 1330 | + return logs; | |
| 1331 | +} | |
| 1332 | + | |
| 1333 | +const toolkit = {}; | |
| 1334 | + | |
| 1335 | +test("official actions: checkout, cache, artifacts, github-script and setup-* run unmodified", { timeout: 1_200_000 }, async () => { | |
| 1336 | + const owner = (await ok("GET", "/user")).login; | |
| 1337 | + toolkit.owner = owner; | |
| 1338 | + // A submodule repository, then the main one (LFS object, submodule, sparse-checkout target). | |
| 1339 | + // Public: like GITHUB_TOKEN, the job token reads only its own repository and public ones. | |
| 1340 | + await ok("POST", "/user/repos", { name: "toolkit-lib", visibility: "public" }); | |
| 1341 | + const lib = join(work, "toolkit-lib"); | |
| 1342 | + mkdirSync(lib, { recursive: true }); | |
| 1343 | + writeFileSync(join(lib, "lib.txt"), "from the submodule\n"); | |
| 1344 | + git(lib, ["init", "-q", "-b", "main"]); | |
| 1345 | + git(lib, ["add", "."]); | |
| 1346 | + git(lib, ["commit", "-q", "-m", "lib"]); | |
| 1347 | + git(lib, ["remote", "add", "origin", `${base}/${owner}/toolkit-lib.git`]); | |
| 1348 | + git(lib, ["push", "-q", "origin", "main"]); | |
| 1349 | + const libSha = git(lib, ["rev-parse", "HEAD"]).trim(); | |
| 1350 | + | |
| 1351 | + await ok("POST", "/user/repos", { name: "toolkit" }); | |
| 1352 | + // Jobs without permissions: write with the default token (the setting older repositories keep). | |
| 1353 | + await ok("PUT", `/repos/${owner}/toolkit/actions/permissions/workflow`, { default_workflow_permissions: "write" }); | |
| 1354 | + const src = join(work, "toolkit"); | |
| 1355 | + toolkit.src = src; | |
| 1356 | + mkdirSync(join(src, ".github/workflows"), { recursive: true }); | |
| 1357 | + mkdirSync(join(src, "docs")); | |
| 1358 | + writeFileSync(join(src, ".github/workflows/toolkit.yml"), TOOLKIT_WORKFLOW); | |
| 1359 | + writeFileSync(join(src, "docs/guide.md"), "# guide\n"); | |
| 1360 | + writeFileSync(join(src, "README.md"), "# toolkit\n"); | |
| 1361 | + writeFileSync(join(src, "lock.txt"), "v1\n"); | |
| 1362 | + writeFileSync(join(src, "package.json"), JSON.stringify({ name: "toolkit", version: "1.0.0", private: true }, null, 2) + "\n"); | |
| 1363 | + writeFileSync(join(src, "package-lock.json"), JSON.stringify({ name: "toolkit", version: "1.0.0", lockfileVersion: 3, requires: true, packages: { "": { name: "toolkit", version: "1.0.0" } } }, null, 2) + "\n"); | |
| 1364 | + writeFileSync(join(src, "pom.xml"), "<project><modelVersion>4.0.0</modelVersion><groupId>e2e</groupId><artifactId>toolkit</artifactId><version>1</version></project>\n"); | |
| 1365 | + writeFileSync(join(src, ".gitmodules"), '[submodule "lib"]\n\tpath = lib\n\turl = ../toolkit-lib.git\n'); | |
| 1366 | + git(src, ["init", "-q", "-b", "main"]); | |
| 1367 | + git(src, ["lfs", "install", "--local"]); | |
| 1368 | + git(src, ["lfs", "track", "*.bin"]); | |
| 1369 | + writeFileSync(join(src, "big.bin"), "the real LFS content\n"); | |
| 1370 | + git(src, ["add", "-A"]); | |
| 1371 | + git(src, ["update-index", "--add", "--cacheinfo", `160000,${libSha},lib`]); | |
| 1372 | + git(src, ["commit", "-q", "-m", "toolkit"]); | |
| 1373 | + git(src, ["remote", "add", "origin", `${base}/${owner}/toolkit.git`]); | |
| 1374 | + git(src, ["push", "-q", "origin", "main"]); | |
| 1375 | + assert.match(git(src, ["show", "HEAD:big.bin"]), /git-lfs/, "big.bin is stored as an LFS pointer"); | |
| 1376 | + | |
| 1377 | + const run = await until("toolkit run", async () => (await ok("GET", `/repos/${owner}/toolkit/actions/runs`)).workflow_runs[0]); | |
| 1378 | + toolkit.firstRun = run; | |
| 1379 | + const logs = await runToolkit(owner, run); | |
| 1380 | + | |
| 1381 | + // checkout: LFS content, submodule over HTTP with the job token, credentials not persisted, sparse | |
| 1382 | + assert.match(logs.produce, /the real LFS content\nfrom the submodule/); | |
| 1383 | + assert.match(logs.produce, /no persisted credentials/); | |
| 1384 | + assert.match(logs.produce, /sparse=README\.md big\.bin docs lib lock\.txt package-lock\.json package\.json pom\.xml /); | |
| 1385 | + // cache: a miss on the first run, for the real action and the built-in alike | |
| 1386 | + assert.match(logs.produce, /real-hit= saved by run 1\b/); | |
| 1387 | + assert.match(logs.produce, /builtin-hit= v3-hit=/); | |
| 1388 | + assert.match(logs.produce, /Cache saved with (the )?key: deps-[0-9a-f]{64}/); | |
| 1389 | + // artifacts: ids from both uploads, the URL points back at this server | |
| 1390 | + assert.match(logs.produce, new RegExp(`ids=\\d+,\\d+ url=${base}/${owner}/toolkit/actions/runs/1/artifacts/\\d+`)); | |
| 1391 | + assert.match( | |
| 1392 | + logs.consume, | |
| 1393 | + /all\/dist-builtin\/one\.txt all\/dist-builtin\/sub\/two\.txt all\/dist-real\/\.hidden all\/dist-real\/one\.txt all\/dist-real\/sub\/two\.txt merged\/\.hidden merged\/one\.txt merged\/sub\/two\.txt one\/one\.txt one\/sub\/two\.txt/, | |
| 1394 | + ); | |
| 1395 | + // github-script: REST calls with the job token | |
| 1396 | + assert.match(logs.script, /script says issue=1 labels=from-run-1 comments=1 pulls=0 event=push repo=[^/]+\/toolkit sender=\S+/); | |
| 1397 | + const issue = await ok("GET", `/repos/${owner}/toolkit/issues/1`); | |
| 1398 | + assert.equal(issue.title, "Opened by run 1"); | |
| 1399 | + assert.deepEqual(issue.labels.map((l) => l.name), ["from-run-1"]); | |
| 1400 | + assert.equal((await ok("GET", `/repos/${owner}/toolkit/issues/1/comments`))[0].body, "commented by github-script"); | |
| 1401 | + const statuses = await ok("GET", `/repos/${owner}/toolkit/commits/${run.head_sha}/statuses`); | |
| 1402 | + assert.equal(statuses.find((s) => s.context === "github-script")?.state, "success", JSON.stringify(statuses)); | |
| 1403 | + // `permissions: { contents: read }`: reading works, issue writes are refused | |
| 1404 | + assert.match(logs.restricted, /restricted says repo=toolkit issue write -> 403: .*issues/); | |
| 1405 | + assert.equal((await ok("GET", `/repos/${owner}/toolkit/issues?state=all`)).filter((i) => i.title === "must not be created").length, 0); | |
| 1406 | + // setup-node (the real action, DIFFINERY_DISABLE_BUILTINS=1) and setup-java: installed, caches saved | |
| 1407 | + assert.match(logs.node, /node=v20\.\d+\.\d+ npm-cache-hit=false/); | |
| 1408 | + assert.match(logs.node, /Cache saved with (the )?key: node-cache-Linux-x64-npm-[0-9a-f]{64}/); | |
| 1409 | + assert.match(logs.java, /java=java version "21[^\n]* maven-cache-hit=false run 1/); | |
| 1410 | + assert.match(logs.java, /Cache saved with (the )?key: setup-java-Linux-x64-maven-[0-9a-f]{64}/); | |
| 1411 | + | |
| 1412 | + // the REST API lists the run's artifacts and caches | |
| 1413 | + const artifacts = await ok("GET", `/repos/${owner}/toolkit/actions/runs/1/artifacts`); | |
| 1414 | + assert.deepEqual(artifacts.artifacts.map((a) => a.name).sort(), ["dist-builtin", "dist-real"]); | |
| 1415 | + const caches = await ok("GET", `/repos/${owner}/toolkit/actions/caches`); | |
| 1416 | + assert.equal(caches.total_count, 3, JSON.stringify(caches)); | |
| 1417 | + const zip = await fetch(`${base}/api/v3/repos/${owner}/toolkit/actions/artifacts/${artifacts.artifacts[0].id}/zip`, { headers: { authorization: `Bearer ${token}` }, redirect: "manual" }); | |
| 1418 | + assert.equal(zip.status, 302); | |
| 1419 | + assert.equal((await fetch(zip.headers.get("location"))).headers.get("content-type"), "application/zip"); | |
| 1420 | +}); | |
| 1421 | + | |
| 1422 | +test("official actions: the second run hits every cache and reads the first run's artifacts", { timeout: 1_200_000 }, async () => { | |
| 1423 | + const { owner, src, firstRun } = toolkit; | |
| 1424 | + assert.ok(firstRun, "the first toolkit run must have completed"); | |
| 1425 | + writeFileSync(join(src, "prev-run.txt"), `${firstRun.run_number}\n`); | |
| 1426 | + git(src, ["add", "prev-run.txt"]); | |
| 1427 | + git(src, ["commit", "-q", "-m", "second run"]); | |
| 1428 | + git(src, ["push", "-q", "origin", "main"]); | |
| 1429 | + const run = await until("second toolkit run", async () => (await ok("GET", `/repos/${owner}/toolkit/actions/runs`)).workflow_runs.find((r) => r.run_number === 2)); | |
| 1430 | + const logs = await runToolkit(owner, run); | |
| 1431 | + | |
| 1432 | + // The entry the real action saved restores with the real action (v4 and v3) and the built-in. | |
| 1433 | + assert.match(logs.produce, /real-hit=true saved by run 1\b/); | |
| 1434 | + assert.match(logs.produce, /builtin-hit=true v3-hit=true/); | |
| 1435 | + assert.match(logs.produce, /Cache hit occurred on the primary key deps-[0-9a-f]{64}, not saving cache\./); | |
| 1436 | + assert.match(logs.node, /npm-cache-hit=true/); | |
| 1437 | + assert.match(logs.java, /maven-cache-hit=true run 2/); | |
| 1438 | + // Artifacts of another run, through the REST API with github-token (real action and built-in). | |
| 1439 | + assert.match(logs.consume, /previous run two prev-builtin\/dist-builtin\/one\.txt prev-builtin\/dist-builtin\/sub\/two\.txt prev-builtin\/dist-real\/\.hidden prev-builtin\/dist-real\/one\.txt prev-builtin\/dist-real\/sub\/two\.txt/); | |
| 1440 | + assert.match(logs.script, /script says issue=2 labels=from-run-2 comments=1/); | |
| 1441 | +}); | |
| 1442 | + | |
| 740 | 1443 | test("D1 sessions: bookmarks for token clients and browsers", async () => { |
| 741 | 1444 | // Token clients get the bookmark in a header and can send it back to read their own writes. |
| 742 | 1445 | const r = await fetch(`${base}/api/v3/user`, { headers: { authorization: `Bearer ${token}` } }); |
modifiedapps/diffinery/test/e2e/email.test.mjs+1−1View file
| @@ -103,7 +103,7 @@ | ||
| 103 | 103 | base = `http://127.0.0.1:${port}`; |
| 104 | 104 | const persist = join(work, "state"); |
| 105 | 105 | execFileSync("pnpm", ["exec", "wrangler", "d1", "migrations", "apply", "diffinery-dev", "--local", "-e", "dev", "--persist-to", persist], { cwd: APP, stdio: "pipe", env: { ...process.env, CI: "1" } }); |
| 106 | - dev = spawn("pnpm", ["exec", "wrangler", "dev", "-e", "dev", "--ip", "127.0.0.1", "--port", String(port), "--persist-to", persist, "--var", `PUBLIC_URL:${base}`, "--var", `MASTER_KEY:${randomBytes(32).toString("base64")}`, "--show-interactive-dev-session=false"], { | |
| 106 | + dev = spawn("pnpm", ["exec", "wrangler", "dev", "-e", "dev", "--ip", "127.0.0.1", "--port", String(port), "--inspector-port", String(await freePort()), "--persist-to", persist, "--var", `PUBLIC_URL:${base}`, "--var", `MASTER_KEY:${randomBytes(32).toString("base64")}`, "--show-interactive-dev-session=false"], { | |
| 107 | 107 | cwd: APP, |
| 108 | 108 | env: { ...process.env, CI: "1", WRANGLER_SEND_METRICS: "false" }, |
| 109 | 109 | stdio: ["ignore", "pipe", "pipe"], |
modifiedapps/diffinery/test/e2e/passkeys.test.mjs+1−1View file
| @@ -100,7 +100,7 @@ | ||
| 100 | 100 | ); |
| 101 | 101 | dev = spawn( |
| 102 | 102 | "pnpm", |
| 103 | - ["exec", "wrangler", "dev", "-e", "dev", "--ip", "127.0.0.1", "--port", String(port), "--persist-to", persist, "--var", `PUBLIC_URL:${base}`, "--var", "DEV_AUTH:", "--show-interactive-dev-session=false"], | |
| 103 | + ["exec", "wrangler", "dev", "-e", "dev", "--ip", "127.0.0.1", "--port", String(port), "--inspector-port", String(await freePort()), "--persist-to", persist, "--var", `PUBLIC_URL:${base}`, "--var", "DEV_AUTH:", "--show-interactive-dev-session=false"], | |
| 104 | 104 | { cwd: APP, env: { ...process.env, CI: "1", WRANGLER_SEND_METRICS: "false" }, stdio: ["ignore", "pipe", "pipe"], detached: true }, |
| 105 | 105 | ); |
| 106 | 106 | dev.stdout.on("data", (d) => (devLog += d)); |
modifiedapps/diffinery/test/e2e/registry.test.mjs+3−1View file
| @@ -125,7 +125,7 @@ | ||
| 125 | 125 | `INSERT INTO org_members (org_id, user_id, role, created_at) VALUES ('o-crew', 'u-olga', 'owner', '${at}'), ('o-crew', 'u-mira', 'member', '${at}'), ('o-crew', 'u-nils', 'member', '${at}')`, |
| 126 | 126 | ]; |
| 127 | 127 | execFileSync("pnpm", ["exec", "wrangler", "d1", "execute", "diffinery-dev", "--local", "-e", "dev", "--persist-to", persist, "--command", seed.join("; ")], { cwd: APP, stdio: "pipe", env: { ...process.env, CI: "1" } }); |
| 128 | - dev = spawn("pnpm", ["exec", "wrangler", "dev", "-e", "dev", "--ip", "127.0.0.1", "--port", String(port), "--persist-to", persist, "--var", `PUBLIC_URL:${base}`, "--var", `MASTER_KEY:${randomBytes(32).toString("base64")}`, "--show-interactive-dev-session=false"], { | |
| 128 | + dev = spawn("pnpm", ["exec", "wrangler", "dev", "-e", "dev", "--ip", "127.0.0.1", "--port", String(port), "--inspector-port", String(await freePort()), "--persist-to", persist, "--var", `PUBLIC_URL:${base}`, "--var", `MASTER_KEY:${randomBytes(32).toString("base64")}`, "--show-interactive-dev-session=false"], { | |
| 129 | 129 | cwd: APP, |
| 130 | 130 | env: { ...process.env, CI: "1", WRANGLER_SEND_METRICS: "false" }, |
| 131 | 131 | stdio: ["ignore", "pipe", "pipe"], |
| @@ -445,6 +445,7 @@ | ||
| 445 | 445 | publish: |
| 446 | 446 | runs-on: [self-hosted, linux] |
| 447 | 447 | permissions: |
| 448 | + contents: read | |
| 448 | 449 | packages: write |
| 449 | 450 | steps: |
| 450 | 451 | - uses: actions/checkout@v4 |
| @@ -453,6 +454,7 @@ | ||
| 453 | 454 | needs: publish |
| 454 | 455 | runs-on: [self-hosted, linux] |
| 455 | 456 | permissions: |
| 457 | + contents: read | |
| 456 | 458 | packages: read |
| 457 | 459 | steps: |
| 458 | 460 | - uses: actions/checkout@v4 |
modifiedapps/diffinery/test/e2e/security.test.mjs+138−3View file
| @@ -13,6 +13,9 @@ | ||
| 13 | 13 | // read-scoped tokens can't write; agents can't change secrets or variables; |
| 14 | 14 | // merges honour linear history, need an explicit admin bypass, and drafts can |
| 15 | 15 | // be marked ready; the exact branch protection rule wins over a pattern. |
| 16 | +// Push protection also covers commits made without git (web editor, REST, MCP). | |
| 17 | +// Repository-limited tokens can't mint wider ones; settings changed through the | |
| 18 | +// API are audited; readers' labels and assignees are dropped. | |
| 16 | 19 | import { spawn, execFileSync } from "node:child_process"; |
| 17 | 20 | import { createHash, randomBytes } from "node:crypto"; |
| 18 | 21 | import { mkdtempSync, rmSync } from "node:fs"; |
| @@ -32,7 +35,7 @@ | ||
| 32 | 35 | const hash = (s) => createHash("sha256").update(s).digest("hex"); |
| 33 | 36 | const tok = (name) => `dfy_${name}_${randomBytes(12).toString("hex")}`; |
| 34 | 37 | const T = { olive: tok("olive"), bot: tok("bot"), rita: tok("rita"), dora: tok("dora"), orphan: tok("orphan"), reader: tok("reader"), ritaAdmin: tok("ritaAdmin"), milo: tok("milo") }; |
| 35 | -const S = { penny: `penny-session-${randomBytes(12).toString("hex")}`, dora: `dora-session-${randomBytes(12).toString("hex")}` }; | |
| 38 | +const S = { penny: `penny-session-${randomBytes(12).toString("hex")}`, dora: `dora-session-${randomBytes(12).toString("hex")}`, olive: `olive-session-${randomBytes(12).toString("hex")}` }; | |
| 36 | 39 | const OWNER = "olive"; |
| 37 | 40 | |
| 38 | 41 | const freePort = () => |
| @@ -60,7 +63,9 @@ | ||
| 60 | 63 | const headers = { "content-type": "application/json" }; |
| 61 | 64 | if (typeof as === "string") headers.authorization = `Bearer ${as}`; |
| 62 | 65 | else if (as?.cookie) headers.cookie = `__Host-dfy_session=${as.cookie}`; |
| 63 | - const res = await fetch(`${base}/api/v3${path}`, { method, headers, body: body === undefined ? undefined : JSON.stringify(body) }); | |
| 66 | + const send = () => fetch(`${base}/api/v3${path}`, { method, headers, body: body === undefined ? undefined : JSON.stringify(body) }); | |
| 67 | + // wrangler dev closes idle keep-alive connections (the audit checks pause for seconds), so a reused one can be gone. | |
| 68 | + const res = await send().catch((e) => (e.cause?.code === "UND_ERR_SOCKET" ? send() : Promise.reject(e))); | |
| 64 | 69 | const text = await res.text(); |
| 65 | 70 | let json = null; |
| 66 | 71 | try { |
| @@ -117,11 +122,12 @@ | ||
| 117 | 122 | `INSERT INTO org_members (org_id, user_id, role, created_at) VALUES ('o-guild', 'u-milo', 'member', '${at}')`, |
| 118 | 123 | session("s1", "u-penny", S.penny), |
| 119 | 124 | session("s2", "u-dora", S.dora), |
| 125 | + session("s3", "u-olive", S.olive), | |
| 120 | 126 | ].join("; "), |
| 121 | 127 | ); |
| 122 | 128 | dev = spawn( |
| 123 | 129 | "pnpm", |
| 124 | - ["exec", "wrangler", "dev", "-e", "dev", "--ip", "127.0.0.1", "--port", String(port), "--persist-to", persist, "--var", `PUBLIC_URL:${base}`, "--var", "DEV_AUTH:", "--var", `MASTER_KEY:${randomBytes(32).toString("base64")}`, "--show-interactive-dev-session=false"], | |
| 130 | + ["exec", "wrangler", "dev", "-e", "dev", "--ip", "127.0.0.1", "--port", String(port), "--inspector-port", String(await freePort()), "--persist-to", persist, "--var", `PUBLIC_URL:${base}`, "--var", "DEV_AUTH:", "--var", `MASTER_KEY:${randomBytes(32).toString("base64")}`, "--show-interactive-dev-session=false"], | |
| 125 | 131 | { cwd: APP, env: { ...process.env, CI: "1", WRANGLER_SEND_METRICS: "false" }, stdio: ["ignore", "pipe", "pipe"], detached: true }, |
| 126 | 132 | ); |
| 127 | 133 | dev.stdout.on("data", (d) => (devLog += d)); |
| @@ -286,7 +292,44 @@ | ||
| 286 | 292 | await status(403, "PUT", `${A}/environments/production`, {}); |
| 287 | 293 | await status(403, "POST", `${A}/releases`, { tag_name: "v1" }); |
| 288 | 294 | await status(200, "GET", `${A}/labels`); |
| 295 | + // Every write is refused, issue and admin routes alike... | |
| 296 | + await status(403, "POST", `${A}/issues`, { title: "new" }); | |
| 297 | + await status(403, "POST", `${A}/issues/${old.number}/comments`, { body: "late" }); | |
| 298 | + await status(403, "PATCH", `${A}/issues/${old.number}`, { title: "renamed" }); | |
| 299 | + await status(403, "PUT", `${A}/branches/main/protection`, { required_status_checks: null, enforce_admins: false, required_pull_request_reviews: null, restrictions: null }); | |
| 300 | + await status(403, "PUT", `${A}/collaborators/rita`, { permission: "push" }); | |
| 301 | + await status(403, "PUT", `${A}/actions/secrets/TOKEN`, { value: "s3cret" }); | |
| 302 | + await status(403, "POST", `${A}/actions/variables`, { name: "X", value: "1" }); | |
| 303 | + await status(403, "POST", `${A}/hooks`, { config: { url: "https://example.com/hook" } }); | |
| 304 | + const mcp = await fetch(`${base}/mcp`, { method: "POST", headers: { authorization: `Bearer ${T.bot}`, "content-type": "application/json", accept: "application/json, text/event-stream" }, body: JSON.stringify({ jsonrpc: "2.0", id: 1, method: "tools/call", params: { name: "create_issue", arguments: { owner: OWNER, repo: "attic", title: "via mcp" } } }) }); | |
| 305 | + assert.equal((await mcp.json()).result.isError, true); | |
| 306 | + // ...but reads, watching and unarchiving still work. | |
| 307 | + await status(200, "GET", `${A}/actions/secrets`); | |
| 308 | + await status(200, "PUT", `${A}/subscription`, { subscribed: true }); | |
| 309 | + assert.equal((await ok("PATCH", A, { archived: false })).archived, false); | |
| 310 | + await status(201, "POST", `${A}/issues`, { title: "after unarchiving" }); | |
| 289 | 311 | |
| 312 | + // ------------------------------------------------------------ renaming a variable onto an existing name is a conflict | |
| 313 | + for (const n of ["ONE", "TWO"]) await status(201, "POST", `${A}/actions/variables`, { name: n, value: n }); | |
| 314 | + await status(409, "PATCH", `${A}/actions/variables/ONE`, { name: "two" }); | |
| 315 | + assert.equal((await ok("GET", `${A}/actions/variables/TWO`)).value, "TWO"); | |
| 316 | + await ok("PUT", `${A}/environments/Staging`, {}); | |
| 317 | + for (const n of ["ONE", "TWO"]) await status(201, "POST", `${A}/environments/Staging/variables`, { name: n, value: n }); | |
| 318 | + await status(409, "PATCH", `${A}/environments/Staging/variables/ONE`, { name: "TWO" }); | |
| 319 | + | |
| 320 | + // ------------------------------------------------------------ deployments: environment names ignore case; only older ones go inactive | |
| 321 | + const dep = async (environment) => (await ok("POST", `${A}/deployments`, { ref: "main", environment, required_contexts: [] })).id; | |
| 322 | + const latest = async (id) => (await ok("GET", `${A}/deployments/${id}/statuses`))[0].state; | |
| 323 | + const d1 = await dep("staging"); | |
| 324 | + await ok("POST", `${A}/deployments/${d1}/statuses`, { state: "success" }); | |
| 325 | + const d2 = await dep("STAGING"); | |
| 326 | + assert.equal((await ok("GET", `${A}/deployments/${d2}`)).environment, "Staging"); | |
| 327 | + await ok("POST", `${A}/deployments/${d2}/statuses`, { state: "success" }); | |
| 328 | + assert.equal(await latest(d1), "inactive"); | |
| 329 | + await ok("POST", `${A}/deployments/${d1}/statuses`, { state: "success" }); | |
| 330 | + assert.equal(await latest(d2), "success", "a newer deployment stays active"); | |
| 331 | + assert.equal((await ok("GET", `${A}/deployments?environment=staging`)).length, 2); | |
| 332 | + | |
| 290 | 333 | // ------------------------------------------------------------ private: false makes a repository public |
| 291 | 334 | await repoWithReadme("flip"); |
| 292 | 335 | const flipped = await ok("PATCH", `/repos/${OWNER}/flip`, { private: false }); |
| @@ -407,6 +450,98 @@ | ||
| 407 | 450 | assert.equal((await ok("PUT", `${R}/pulls/${draft.number}/merge`, { merge_method: "squash" })).merged, true); |
| 408 | 451 | }); |
| 409 | 452 | |
| 453 | +test("security: push protection covers commits made without git", async () => { | |
| 454 | + const R = `/repos/${OWNER}/leaky`; | |
| 455 | + const mainSha = await repoWithReadme("leaky"); | |
| 456 | + // Built at run time so this file holds no credential-shaped string. | |
| 457 | + const leak = `AWS_ACCESS_KEY_ID=${"AKIA"}IOSFODNN7EXAMPL3\n`; | |
| 458 | + const files = [{ path: "config.env", content: leak }]; | |
| 459 | + const head = async () => (await ok("GET", `${R}/branches/main`)).commit.sha; | |
| 460 | + | |
| 461 | + // ------------------------------------------------------------ REST: the contents API and the multi-file commits API | |
| 462 | + const viaContents = await status(422, "PUT", `${R}/contents/config.env`, { message: "leak", content: b64(leak) }); | |
| 463 | + assert.match(viaContents.message, /push protection.*AWS access key/); | |
| 464 | + const viaCommits = await status(422, "POST", `${R}/diffinery/commits`, { branch: "main", message: "leak", files }); | |
| 465 | + assert.match(viaCommits.message, /push protection.*AWS access key.*config\.env/); | |
| 466 | + await status(422, "POST", `${R}/diffinery/commits`, { branch: "main", message: "leak", files: [{ path: "k.env", encoding: "base64", content: b64(leak) }] }); | |
| 467 | + // Agents can't bypass, even with a reason; a human admin can, and it's on the record. | |
| 468 | + const agentTry = await status(422, "POST", `${R}/diffinery/commits`, { branch: "main", message: "leak", files, allow_secrets: "please" }, T.bot); | |
| 469 | + assert.match(agentTry.message, /people, not agents/); | |
| 470 | + assert.equal(await head(), mainSha); | |
| 471 | + | |
| 472 | + // ------------------------------------------------------------ MCP commit_files | |
| 473 | + const rpc = async (method, params) => { | |
| 474 | + const r = await fetch(`${base}/mcp`, { method: "POST", headers: { authorization: `Bearer ${T.bot}`, "content-type": "application/json", accept: "application/json, text/event-stream" }, body: JSON.stringify({ jsonrpc: "2.0", id: 1, method, params }) }); | |
| 475 | + assert.equal(r.status, 200); | |
| 476 | + return (await r.json()).result; | |
| 477 | + }; | |
| 478 | + const viaMcp = await rpc("tools/call", { name: "commit_files", arguments: { owner: OWNER, repo: "leaky", branch: "agent-leak", create_branch_from: "main", message: "leak", files } }); | |
| 479 | + assert.equal(viaMcp.isError, true, JSON.stringify(viaMcp)); | |
| 480 | + assert.match(JSON.stringify(viaMcp), /push protection/); | |
| 481 | + await status(404, "GET", `${R}/branches/agent-leak`); | |
| 482 | + | |
| 483 | + // ------------------------------------------------------------ the web editor | |
| 484 | + const cookie = `__Host-dfy_session=${S.olive}`; | |
| 485 | + const page = await fetch(`${base}/${OWNER}/leaky/new/main`, { headers: { cookie } }); | |
| 486 | + assert.equal(page.status, 200); | |
| 487 | + const csrf = /name="_csrf" value="([^"]+)"/.exec(await page.text())?.[1]; | |
| 488 | + const csrfCookie = /dfy_csrf=([^;]+)/.exec(page.headers.get("set-cookie") ?? "")?.[1]; | |
| 489 | + assert.ok(csrf && csrfCookie); | |
| 490 | + const form = new URLSearchParams({ _csrf: csrf, path: "config.env", content: leak, message: "leak", description: "", choice: "direct", new_branch: "", head: mainSha }); | |
| 491 | + const posted = await fetch(`${base}/${OWNER}/leaky/new/main`, { method: "POST", body: form, redirect: "manual", headers: { cookie: `${cookie}; dfy_csrf=${csrfCookie}`, "sec-fetch-site": "same-origin" } }); | |
| 492 | + assert.equal(posted.status, 422); | |
| 493 | + assert.match(await posted.text(), /push protection/); | |
| 494 | + assert.equal(await head(), mainSha); | |
| 495 | + | |
| 496 | + // ------------------------------------------------------------ an admin's explicit bypass goes through | |
| 497 | + const bypassed = await ok("POST", `${R}/diffinery/commits`, { branch: "main", message: "test fixture", files, allow_secrets: "fake key used by this test" }); | |
| 498 | + assert.notEqual(bypassed.sha, mainSha); | |
| 499 | +}); | |
| 500 | + | |
| 501 | +/** Audit entries recorded so far, read straight from the local D1 database. */ | |
| 502 | +function auditLog(action) { | |
| 503 | + const out = execFileSync("pnpm", ["exec", "wrangler", "d1", "execute", "diffinery-dev", "--local", "-e", "dev", "--persist-to", join(work, "state"), "--json", "--command", `SELECT actor_login, action, target, detail FROM audit_log WHERE action = '${action}'`], { cwd: APP, stdio: "pipe", env: { ...process.env, CI: "1" } }); | |
| 504 | + return JSON.parse(out.toString())[0].results; | |
| 505 | +} | |
| 506 | + | |
| 507 | +test("security: limited tokens, invisible repositories, readers' labels and the audit trail", async () => { | |
| 508 | + await repoWithReadme("limits"); | |
| 509 | + // ------------------------------------------------------------ a token limited to some repositories can't mint one that isn't | |
| 510 | + const limited = (await ok("POST", "/diffinery/tokens", { name: "limited", scopes: ["admin"], repositories: [`${OWNER}/limits`] })).token; | |
| 511 | + await status(403, "POST", "/diffinery/tokens", { name: "wider", scopes: ["admin"] }, limited); | |
| 512 | + // ------------------------------------------------------------ private repositories you can't see stay unknown | |
| 513 | + await ok("POST", "/user/repos", { name: "hidden", private: true }, T.rita); | |
| 514 | + await status(422, "POST", "/diffinery/tokens", { name: "peek", repositories: ["rita/hidden"] }); | |
| 515 | + await status(404, "POST", "/repos/rita/hidden/issues", { title: "hi" }, T.reader); | |
| 516 | + // An anonymous push is asked for credentials (git only sends them after a 401), even to a public repository. | |
| 517 | + await ok("PATCH", `/repos/${OWNER}/limits`, { visibility: "public" }); | |
| 518 | + const push = await fetch(`${base}/${OWNER}/limits.git/info/refs?service=git-receive-pack`); | |
| 519 | + assert.equal(push.status, 401); | |
| 520 | + assert.match(push.headers.get("www-authenticate") ?? "", /Basic/); | |
| 521 | + assert.equal((await fetch(`${base}/${OWNER}/limits.git/info/refs?service=git-receive-pack`, { headers: { authorization: `Basic ${Buffer.from(`x:${T.olive}`).toString("base64")}` } })).status, 200); | |
| 522 | + | |
| 523 | + // ------------------------------------------------------------ labels and assignees from readers are dropped, as on GitHub | |
| 524 | + const R = `/repos/${OWNER}/limits`; | |
| 525 | + await ok("POST", `${R}/labels`, { name: "bug", color: "ff0000" }); | |
| 526 | + const fromReader = await ok("POST", `${R}/issues`, { title: "from a reader", labels: ["bug"], assignees: [OWNER] }, T.rita); | |
| 527 | + assert.deepEqual([fromReader.labels.length, fromReader.assignees.length], [0, 0]); | |
| 528 | + const fromAdmin = await ok("POST", `${R}/issues`, { title: "from the owner", labels: ["bug"] }); | |
| 529 | + assert.equal(fromAdmin.labels[0].name, "bug"); | |
| 530 | + | |
| 531 | + // ------------------------------------------------------------ settings changed through the API are on the record | |
| 532 | + await status(204, "PUT", `${R}/collaborators/rita`, { permission: "push" }); | |
| 533 | + await ok("PUT", `${R}/branches/main/protection`, { requiredApprovals: 1 }); | |
| 534 | + await status(204, "PUT", `${R}/actions/secrets/TOKEN`, { value: "s3cret" }); | |
| 535 | + await status(201, "POST", `${R}/actions/variables`, { name: "MODE", value: "x" }); | |
| 536 | + await status(201, "POST", `${R}/hooks`, { config: { url: "https://example.com/hook" } }); | |
| 537 | + await until("audit entries", async () => { | |
| 538 | + const want = ["repo.update", "repo.collaborator.set", "repo.protection.set", "repo.secret.set", "repo.variable.set", "repo.webhook.create"]; | |
| 539 | + return want.every((a) => auditLog(a).some((e) => e.target === `${OWNER}/limits` && JSON.parse(e.detail).via === "api")); | |
| 540 | + }); | |
| 541 | + const leak = auditLog("api.blocked_secret").find((e) => e.target === `${OWNER}/leaky`); | |
| 542 | + assert.equal(leak?.actor_login, OWNER, "blocked API writes are audited against the caller"); | |
| 543 | +}); | |
| 544 | + | |
| 410 | 545 | test("security: organization pictures are the owners' to change, and invitations aren't sent twice", async () => { |
| 411 | 546 | // rita is an admin of guild: like its name, the picture is for owners (the settings page agrees). |
| 412 | 547 | const denied = await status(403, "DELETE", "/orgs/guild/avatar", undefined, T.ritaAdmin); |
addedapps/diffinery/test/unit/actions-runtime.test.ts+517−0View file
| @@ -1,0 +1,517 @@ | ||
| 1 | +import { describe, expect, it } from "vitest"; | |
| 2 | +import { BodyTooLarge, STEP_OPS, batchParts, blockKey, chunkKey, compose, finalKey, listStaged, parseBlockList, parseRange, planCompose, putStream, runComposeStep, stagingPrefix, type BlobBucket, type ComposeLimits, type ComposeStep } from "../../src/lib/blob-store.js"; | |
| 3 | +import { DatabaseSync } from "node:sqlite"; | |
| 4 | +import { Runtime } from "../../src/repo/runtime.js"; | |
| 5 | +import { REPO_SCHEMA, migrateRepoSchema } from "../../src/repo/schema.js"; | |
| 6 | +import type { RepoDO } from "../../src/repo/RepoDO.js"; | |
| 7 | +import type { Env } from "../../src/env.js"; | |
| 8 | +import { archiveKey, deletePrefixes, legacyArtifactKeys, legacyCacheKeys, logKeys, transientPrefixes } from "../../src/lib/storage-keys.js"; | |
| 9 | +import { signBlobUrl, signRuntimeToken, verifyBlobUrl, verifyRuntimeToken } from "../../src/lib/runtime-auth.js"; | |
| 10 | +import { b64, fromB64url } from "../../src/lib/util.js"; | |
| 11 | +import { completePayload, issuePayload, releasePayload } from "../../src/repo/event-payloads.js"; | |
| 12 | +import type { RepoMeta } from "../../src/repo/types.js"; | |
| 13 | + | |
| 14 | +const MASTER = b64(new Uint8Array(32).fill(7)); | |
| 15 | +const OTHER = b64(new Uint8Array(32).fill(9)); | |
| 16 | +const soon = () => Math.floor(Date.now() / 1000) + 600; | |
| 17 | + | |
| 18 | +describe("runtime tokens", () => { | |
| 19 | + it("carries what the toolkit and buildkit read, and verifies only under the right key", async () => { | |
| 20 | + const token = await signRuntimeToken(MASTER, "https://d.example", { repo: "r1", run: "run1", job: "job1", exp: soon() }, ["refs/heads/feat", "refs/heads/main"]); | |
| 21 | + const claims = JSON.parse(new TextDecoder().decode(fromB64url(token.split(".")[1]!))); | |
| 22 | + // @actions/artifact takes the run and job backend ids from scp. | |
| 23 | + expect(claims.scp).toContain("Actions.Results:run1:job1"); | |
| 24 | + // go-actions-cache needs ac (a JSON string), exp and nbf. | |
| 25 | + expect(JSON.parse(claims.ac)).toEqual([ | |
| 26 | + { Scope: "refs/heads/feat", Permission: 3 }, | |
| 27 | + { Scope: "refs/heads/main", Permission: 1 }, | |
| 28 | + ]); | |
| 29 | + expect(claims.nbf).toBeLessThanOrEqual(Math.floor(Date.now() / 1000)); | |
| 30 | + expect(await verifyRuntimeToken(MASTER, token)).toMatchObject({ repo: "r1", run: "run1", job: "job1" }); | |
| 31 | + expect(await verifyRuntimeToken(OTHER, token)).toBeNull(); | |
| 32 | + expect(await verifyRuntimeToken(MASTER, token.slice(0, -2) + "xx")).toBeNull(); | |
| 33 | + expect(await verifyRuntimeToken(MASTER, "not.a.token")).toBeNull(); | |
| 34 | + }); | |
| 35 | + | |
| 36 | + it("rejects expired tokens and tampered claims", async () => { | |
| 37 | + const expired = await signRuntimeToken(MASTER, "x", { repo: "r", run: "a", job: "b", exp: Math.floor(Date.now() / 1000) - 5 }, ["refs/heads/main"]); | |
| 38 | + expect(await verifyRuntimeToken(MASTER, expired)).toBeNull(); | |
| 39 | + const token = await signRuntimeToken(MASTER, "x", { repo: "r", run: "a", job: "b", exp: soon() }, ["refs/heads/main"]); | |
| 40 | + const [h, p, s] = token.split("."); | |
| 41 | + const forged = JSON.parse(new TextDecoder().decode(fromB64url(p!))); | |
| 42 | + forged.repo = "victim"; | |
| 43 | + const p2 = Buffer.from(JSON.stringify(forged)).toString("base64url"); | |
| 44 | + expect(await verifyRuntimeToken(MASTER, `${h}.${p2}.${s}`)).toBeNull(); | |
| 45 | + }); | |
| 46 | +}); | |
| 47 | + | |
| 48 | +describe("signed blob URLs", () => { | |
| 49 | + it("grant one permission on one blob until they expire", async () => { | |
| 50 | + const ref = { repo: "r1", kind: "artifact" as const, id: 42 }; | |
| 51 | + const url = new URL(await signBlobUrl(MASTER, "https://d.example", ref, "r", soon(), "my artifact.zip")); | |
| 52 | + expect(url.pathname).toBe("/_runtime/blob/r1/artifact/42/my%20artifact.zip"); | |
| 53 | + expect(await verifyBlobUrl(MASTER, ref, "r", url.searchParams)).toBe(true); | |
| 54 | + expect(await verifyBlobUrl(MASTER, ref, "w", url.searchParams)).toBe(false); | |
| 55 | + expect(await verifyBlobUrl(MASTER, { ...ref, id: 43 }, "r", url.searchParams)).toBe(false); | |
| 56 | + expect(await verifyBlobUrl(MASTER, { ...ref, kind: "cache" }, "r", url.searchParams)).toBe(false); | |
| 57 | + expect(await verifyBlobUrl(OTHER, ref, "r", url.searchParams)).toBe(false); | |
| 58 | + const stale = new URL(await signBlobUrl(MASTER, "https://d.example", ref, "r", Math.floor(Date.now() / 1000) - 1)); | |
| 59 | + expect(await verifyBlobUrl(MASTER, ref, "r", stale.searchParams)).toBe(false); | |
| 60 | + url.searchParams.set("se", String(soon() + 100)); | |
| 61 | + expect(await verifyBlobUrl(MASTER, ref, "r", url.searchParams)).toBe(false); | |
| 62 | + }); | |
| 63 | +}); | |
| 64 | + | |
| 65 | +describe("blob protocol helpers", () => { | |
| 66 | + it("parses Azure block lists in order", () => { | |
| 67 | + const xml = `<?xml version="1.0" encoding="utf-8"?><BlockList><Latest>AAA=</Latest><Uncommitted>Qg==</Uncommitted><Committed>a&b</Committed></BlockList>`; | |
| 68 | + expect(parseBlockList(xml)).toEqual(["AAA=", "Qg==", "a&b"]); | |
| 69 | + }); | |
| 70 | + | |
| 71 | + it("parses byte ranges like Azure and HTTP", () => { | |
| 72 | + expect(parseRange(null, 10)).toBeNull(); | |
| 73 | + expect(parseRange("bytes=0-", 10)).toEqual({ offset: 0, length: 10 }); | |
| 74 | + expect(parseRange("bytes=2-4", 10)).toEqual({ offset: 2, length: 3 }); | |
| 75 | + expect(parseRange("bytes=5-100", 10)).toEqual({ offset: 5, length: 5 }); | |
| 76 | + expect(parseRange("bytes=-3", 10)).toEqual({ offset: 7, length: 3 }); | |
| 77 | + expect(parseRange("bytes=10-", 10)).toBe("invalid"); | |
| 78 | + expect(parseRange("bytes=4-2", 10)).toBe("invalid"); | |
| 79 | + expect(parseRange("items=0-1", 10)).toBe("invalid"); | |
| 80 | + }); | |
| 81 | + | |
| 82 | + it("keeps staged objects per blob, and transient data outside repos/", () => { | |
| 83 | + const a = { repo: "r", kind: "cache" as const, id: 1 }; | |
| 84 | + expect(finalKey(a)).toBe("actions-cache/r/1"); | |
| 85 | + expect(finalKey({ ...a, kind: "artifact" })).toBe("artifacts/r/1"); | |
| 86 | + expect(blockKey(a, "x/y")).toMatch(/^actions-staging\/r\/cache-1\/b-[A-Za-z0-9_-]+$/); | |
| 87 | + expect(chunkKey(a, 1024)).toBe("actions-staging/r/cache-1/o-0000000000001024"); | |
| 88 | + // Old keys are still read, after the new ones. | |
| 89 | + expect(logKeys("r", "j")).toEqual(["logs/r/j.json", "repos/r/logs/j.json"]); | |
| 90 | + expect(legacyCacheKeys("r", "a b")).toEqual(["actions-cache/r/legacy/a%20b", "repos/r/cache/a%20b"]); | |
| 91 | + expect(legacyArtifactKeys("r", "run", "n")).toEqual(["artifacts/r/legacy/run/n", "repos/r/artifacts/run/n"]); | |
| 92 | + // Nothing transient is written under repos/, which has no age rule (git objects, LFS). | |
| 93 | + for (const k of [finalKey(a), finalKey({ ...a, kind: "artifact" }), blockKey(a, "b"), logKeys("r", "j")[0], legacyCacheKeys("r", "k")[0], legacyArtifactKeys("r", "x", "y")[0]]) expect(k.startsWith("repos/")).toBe(false); | |
| 94 | + }); | |
| 95 | +}); | |
| 96 | + | |
| 97 | +/** An in-memory stand-in for the R2 bucket, enforcing R2's multipart rules and counting calls. */ | |
| 98 | +function memoryBucket() { | |
| 99 | + const objects = new Map<string, { data: Uint8Array; etag: string }>(); | |
| 100 | + const uploads = new Map<string, { key: string; parts: Map<number, Uint8Array> }>(); | |
| 101 | + const stats = { multipart: 0, parts: [] as number[], ops: 0 }; | |
| 102 | + let n = 0; | |
| 103 | + const bytes = async (v: ReadableStream | ArrayBuffer | ArrayBufferView) => | |
| 104 | + v instanceof ReadableStream ? new Uint8Array(await new Response(v).arrayBuffer()) : v instanceof ArrayBuffer ? new Uint8Array(v) : new Uint8Array(v.buffer, v.byteOffset, v.byteLength).slice(); | |
| 105 | + const store = (key: string, data: Uint8Array) => { | |
| 106 | + const etag = `e${++n}`; | |
| 107 | + objects.set(key, { data, etag }); | |
| 108 | + return { size: data.length, etag }; | |
| 109 | + }; | |
| 110 | + const handle = (uploadId: string) => { | |
| 111 | + const u = uploads.get(uploadId)!; | |
| 112 | + return { | |
| 113 | + uploadId, | |
| 114 | + async uploadPart(p: number, value: ReadableStream | ArrayBuffer | ArrayBufferView) { | |
| 115 | + stats.ops++; | |
| 116 | + u.parts.set(p, await bytes(value)); | |
| 117 | + return { partNumber: p, etag: `p${p}` }; | |
| 118 | + }, | |
| 119 | + async complete(list: Array<{ partNumber: number; etag: string }>) { | |
| 120 | + stats.ops++; | |
| 121 | + const sizes = list.map((p) => u.parts.get(p.partNumber)!.length); | |
| 122 | + sizes.slice(0, -1).forEach((s) => expect(s).toBeGreaterThanOrEqual(5 * 1024 * 1024)); | |
| 123 | + sizes.slice(0, -1).forEach((s) => expect(s).toBe(sizes[0])); | |
| 124 | + expect(list.map((p) => p.partNumber)).toEqual(list.map((_, i) => i + 1)); | |
| 125 | + stats.parts = sizes; | |
| 126 | + const out = new Uint8Array(sizes.reduce((a, b) => a + b, 0)); | |
| 127 | + let at = 0; | |
| 128 | + for (const p of list) { | |
| 129 | + out.set(u.parts.get(p.partNumber)!, at); | |
| 130 | + at += u.parts.get(p.partNumber)!.length; | |
| 131 | + } | |
| 132 | + store(u.key, out); | |
| 133 | + return {}; | |
| 134 | + }, | |
| 135 | + async abort() { | |
| 136 | + stats.ops++; | |
| 137 | + uploads.delete(uploadId); | |
| 138 | + }, | |
| 139 | + }; | |
| 140 | + }; | |
| 141 | + const bucket: BlobBucket = { | |
| 142 | + async get(key, o) { | |
| 143 | + stats.ops++; | |
| 144 | + const obj = objects.get(key); | |
| 145 | + if (!obj) return null; | |
| 146 | + const data = o?.range ? obj.data.subarray(o.range.offset, o.range.offset + o.range.length) : obj.data; | |
| 147 | + return { body: new Response(data).body!, size: data.length }; | |
| 148 | + }, | |
| 149 | + async head(key) { | |
| 150 | + stats.ops++; | |
| 151 | + const o = objects.get(key); | |
| 152 | + return o ? { size: o.data.length, etag: o.etag } : null; | |
| 153 | + }, | |
| 154 | + async put(key, value) { | |
| 155 | + stats.ops++; | |
| 156 | + return store(key, await bytes(value)); | |
| 157 | + }, | |
| 158 | + async list({ prefix, cursor, limit }) { | |
| 159 | + stats.ops++; | |
| 160 | + const all = [...objects].filter(([k]) => k.startsWith(prefix)).map(([key, v]) => ({ key, size: v.data.length })); | |
| 161 | + const start = Number(cursor ?? 0); | |
| 162 | + const page = all.slice(start, start + (limit ?? 1000)); | |
| 163 | + const truncated = start + page.length < all.length; | |
| 164 | + return { objects: page, truncated, cursor: truncated ? String(start + page.length) : undefined }; | |
| 165 | + }, | |
| 166 | + async delete(keys) { | |
| 167 | + stats.ops++; | |
| 168 | + for (const k of Array.isArray(keys) ? keys : [keys]) objects.delete(k); | |
| 169 | + }, | |
| 170 | + async createMultipartUpload(key) { | |
| 171 | + stats.ops++; | |
| 172 | + stats.multipart++; | |
| 173 | + const id = `u${++n}`; | |
| 174 | + uploads.set(id, { key, parts: new Map() }); | |
| 175 | + return handle(id); | |
| 176 | + }, | |
| 177 | + resumeMultipartUpload(_key, uploadId) { | |
| 178 | + return handle(uploadId); | |
| 179 | + }, | |
| 180 | + }; | |
| 181 | + return { bucket, objects, stats }; | |
| 182 | +} | |
| 183 | + | |
| 184 | +const filled = (n: number, seed: number) => { | |
| 185 | + const b = new Uint8Array(n); | |
| 186 | + for (let i = 0; i < n; i++) b[i] = (i * 31 + seed) & 255; | |
| 187 | + return b; | |
| 188 | +}; | |
| 189 | + | |
| 190 | +describe("compose", () => { | |
| 191 | + const ref = { repo: "r", kind: "artifact" as const, id: 9 }; | |
| 192 | + const MiB = 1024 * 1024; | |
| 193 | + | |
| 194 | + /** | |
| 195 | + * Composes blocks of `sizes`, running every step as production does (an invocation | |
| 196 | + * of its own), and checks the bytes. Returns the calls the Worker made and the most | |
| 197 | + * any single step made. | |
| 198 | + */ | |
| 199 | + async function run(sizes: number[], limits?: ComposeLimits) { | |
| 200 | + const m = memoryBucket(); | |
| 201 | + const segs = []; | |
| 202 | + for (const [i, n] of sizes.entries()) { | |
| 203 | + const key = blockKey(ref, `block-${i}`); | |
| 204 | + await m.bucket.put(key, filled(n, i)); | |
| 205 | + segs.push({ key, size: n }); | |
| 206 | + } | |
| 207 | + const staged = await listStaged(m.bucket, ref); | |
| 208 | + expect(staged.size).toBe(sizes.length); | |
| 209 | + m.stats.ops = 0; | |
| 210 | + let steps = 0; | |
| 211 | + let maxStepOps = 0; | |
| 212 | + const runStep = async (step: ComposeStep) => { | |
| 213 | + steps++; | |
| 214 | + const before = m.stats.ops; | |
| 215 | + const out = await runComposeStep(m.bucket, step); | |
| 216 | + const used = m.stats.ops - before; | |
| 217 | + maxStepOps = Math.max(maxStepOps, used); | |
| 218 | + // the Worker's own calls exclude the steps' | |
| 219 | + m.stats.ops = before; | |
| 220 | + return out; | |
| 221 | + }; | |
| 222 | + const total = await compose(m.bucket, finalKey(ref), segs, "application/zip", { scratch: stagingPrefix(ref), run: runStep, concurrency: 1, limits }); | |
| 223 | + const out = m.objects.get(finalKey(ref))!.data; | |
| 224 | + const expected = new Uint8Array(total); | |
| 225 | + let at = 0; | |
| 226 | + for (const [i, n] of sizes.entries()) { | |
| 227 | + expected.set(filled(n, i), at); | |
| 228 | + at += n; | |
| 229 | + } | |
| 230 | + expect(out.length).toBe(expected.length); | |
| 231 | + expect(Buffer.from(out).equals(Buffer.from(expected))).toBe(true); | |
| 232 | + // intermediates are removed; only the blocks (deleted by the caller) and the result remain | |
| 233 | + expect([...m.objects.keys()].filter((k) => !staged.has(k) && k !== finalKey(ref))).toEqual([]); | |
| 234 | + return { ...m.stats, workerOps: m.stats.ops, steps, maxStepOps }; | |
| 235 | + } | |
| 236 | + | |
| 237 | + it("handles empty and single uploads", async () => { | |
| 238 | + expect((await run([])).multipart).toBe(0); | |
| 239 | + expect((await run([123])).multipart).toBe(0); | |
| 240 | + }); | |
| 241 | + | |
| 242 | + it("streams a few blocks into one object", async () => { | |
| 243 | + const r = await run([1000, 7, 1 << 20, 8 * MiB]); | |
| 244 | + expect(r.multipart).toBe(0); | |
| 245 | + expect(r.steps).toBe(1); | |
| 246 | + }); | |
| 247 | + | |
| 248 | + it("writes large uploads as equal multipart parts, cutting across blocks", async () => { | |
| 249 | + const small = { stepOps: 400, maxMergeBytes: 4 * MiB }; | |
| 250 | + expect((await run([8 * MiB, 8 * MiB, 3 * MiB], small)).parts).toEqual([8 * MiB, 8 * MiB, 3 * MiB]); | |
| 251 | + // Go SDK style 1 MiB blocks are re-cut into 8 MiB parts (ranged reads across block boundaries) | |
| 252 | + expect((await run(Array.from({ length: 19 }, (_, i) => (i === 18 ? 12345 : MiB)), small)).parts).toEqual([8 * MiB, 8 * MiB, 2 * MiB + 12345]); | |
| 253 | + expect((await run([3 * MiB, 7 * MiB, 5 * MiB, 1], small)).parts).toEqual([8 * MiB, 7 * MiB + 1]); | |
| 254 | + }); | |
| 255 | + | |
| 256 | + it("merges many small blocks in rounds so that no step exceeds its calls", async () => { | |
| 257 | + // 10 calls per step: 300 blocks need two rounds of merging before the parts | |
| 258 | + const r = await run(Array.from({ length: 300 }, (_, i) => 40 * 1024 + i), { stepOps: 10, maxMergeBytes: 1 * MiB }); | |
| 259 | + expect(r.maxStepOps).toBeLessThanOrEqual(10); | |
| 260 | + expect(r.multipart).toBe(1); | |
| 261 | + }); | |
| 262 | + | |
| 263 | + it("keeps a 50,000-block list (Azure's maximum) within the Workers subrequest limit", async () => { | |
| 264 | + // Real data: 50k tiny blocks, as buildkit's gha cache can send. | |
| 265 | + const r = await run(Array.from({ length: 50_000 }, (_, i) => 16 + (i % 7))); | |
| 266 | + expect(r.maxStepOps).toBeLessThanOrEqual(STEP_OPS); | |
| 267 | + expect(r.steps + r.workerOps).toBeLessThan(1000); | |
| 268 | + // Planned only (no bytes): 50k blocks of 200 KiB (10 GB), and of 1 MiB with 4 MiB blocks mixed in. | |
| 269 | + for (const sizes of [Array.from({ length: 50_000 }, () => 200 * 1024), Array.from({ length: 50_000 }, (_, i) => (i % 10 ? MiB : 4 * MiB))]) { | |
| 270 | + const plan = planCompose(sizes.map((size, i) => ({ key: `k${i}`, size })), "s/"); | |
| 271 | + const steps = [...plan.rounds.flat(), ...batchParts(plan.parts)]; | |
| 272 | + for (const s of steps) { | |
| 273 | + const ops = "kind" in s ? s.slices.length + 1 : s.reduce((n, p) => n + p.slices.length + 1, 0); | |
| 274 | + expect(ops).toBeLessThanOrEqual(STEP_OPS); | |
| 275 | + } | |
| 276 | + // list pages and deletes of 50k staged blocks (50 each), the steps, create and complete | |
| 277 | + expect(50 + 50 + steps.length + 2).toBeLessThan(1000); | |
| 278 | + expect(plan.parts.length).toBeLessThanOrEqual(10_000); | |
| 279 | + expect(plan.parts.slice(0, -1).every((p) => p.reduce((n, s) => n + s.length, 0) === plan.partSize)).toBe(true); | |
| 280 | + } | |
| 281 | + }); | |
| 282 | +}); | |
| 283 | + | |
| 284 | +describe("deleting a repository's transient objects", () => { | |
| 285 | + it("removes every object under its prefixes, a page at a time, and nothing else", async () => { | |
| 286 | + const m = memoryBucket(); | |
| 287 | + const keys = [...Array.from({ length: 2500 }, (_, i) => `actions-cache/r1/${i}`), "artifacts/r1/1", "logs/r1/j.json", "actions-staging/r1/cache-1/b-x", archiveKey("r1", "abc", "zip")]; | |
| 288 | + for (const k of keys) await m.bucket.put(k, new Uint8Array(1)); | |
| 289 | + for (const k of ["actions-cache/r10/1", "repos/r1/objects/ab/cd", "artifacts/r2/1"]) await m.bucket.put(k, new Uint8Array(1)); | |
| 290 | + m.stats.ops = 0; | |
| 291 | + expect(transientPrefixes("r1").some((p) => p.startsWith("repos/"))).toBe(false); | |
| 292 | + expect(await deletePrefixes(m.bucket, transientPrefixes("r1"))).toBe(true); | |
| 293 | + expect([...m.objects.keys()].sort()).toEqual(["actions-cache/r10/1", "artifacts/r2/1", "repos/r1/objects/ab/cd"]); | |
| 294 | + // 3 pages of caches (list + delete each), then a page for each other prefix | |
| 295 | + expect(m.stats.ops).toBe(6 + 4 * 2); | |
| 296 | + // stops within its budget and says so | |
| 297 | + for (let i = 0; i < 3000; i++) await m.bucket.put(`logs/r3/${i}`, new Uint8Array(1)); | |
| 298 | + expect(await deletePrefixes(m.bucket, transientPrefixes("r3"), 4)).toBe(false); | |
| 299 | + }); | |
| 300 | +}); | |
| 301 | + | |
| 302 | +describe("streaming bodies of unknown length", () => { | |
| 303 | + const stream = (chunks: number[]) => | |
| 304 | + new ReadableStream<Uint8Array>({ | |
| 305 | + start(ctl) { | |
| 306 | + chunks.forEach((n, i) => ctl.enqueue(filled(n, i))); | |
| 307 | + ctl.close(); | |
| 308 | + }, | |
| 309 | + }); | |
| 310 | + | |
| 311 | + it("stores small bodies in one put and large ones as parts, a part in memory at a time", async () => { | |
| 312 | + const m = memoryBucket(); | |
| 313 | + expect(await putStream(m.bucket, "a", stream([10, 20]), 100)).toBe(30); | |
| 314 | + expect(m.stats.multipart).toBe(0); | |
| 315 | + const MiB = 1024 * 1024; | |
| 316 | + expect(await putStream(m.bucket, "b", stream([3 * MiB, 3 * MiB, 3 * MiB, 3 * MiB, 1]), 64 * MiB, undefined, 5 * MiB)).toBe(12 * MiB + 1); | |
| 317 | + expect(m.stats.parts).toEqual([5 * MiB, 5 * MiB, 2 * MiB + 1]); | |
| 318 | + expect(m.objects.get("b")!.data.length).toBe(12 * MiB + 1); | |
| 319 | + }); | |
| 320 | + | |
| 321 | + it("refuses more than the limit and stores nothing", async () => { | |
| 322 | + const m = memoryBucket(); | |
| 323 | + await expect(putStream(m.bucket, "c", stream([60, 60]), 100)).rejects.toBeInstanceOf(BodyTooLarge); | |
| 324 | + await expect(putStream(m.bucket, "d", stream([6 * 1024 * 1024, 6 * 1024 * 1024]), 10 * 1024 * 1024, undefined, 5 * 1024 * 1024)).rejects.toBeInstanceOf(BodyTooLarge); | |
| 325 | + expect(m.objects.has("c")).toBe(false); | |
| 326 | + expect(m.objects.has("d")).toBe(false); | |
| 327 | + }); | |
| 328 | +}); | |
| 329 | + | |
| 330 | +describe("event payloads", () => { | |
| 331 | + const meta: RepoMeta = { repoId: "r1", ownerId: "o1", owner: "acme", name: "app", defaultBranch: "main", ownerKind: "org" }; | |
| 332 | + const site = { publicUrl: "https://d.example", meta }; | |
| 333 | + | |
| 334 | + it("adds repository, sender and organization like GitHub", () => { | |
| 335 | + const p = completePayload(site, { ref: "refs/heads/main", sender: { login: "ada" } }, { id: "u1", login: "ada", kind: "user" }); | |
| 336 | + const repo = p.repository as Record<string, unknown>; | |
| 337 | + expect(repo).toMatchObject({ name: "app", full_name: "acme/app", default_branch: "main", clone_url: "https://d.example/acme/app.git", url: "https://d.example/api/v3/repos/acme/app" }); | |
| 338 | + expect(repo.owner).toMatchObject({ login: "acme", type: "Organization" }); | |
| 339 | + expect(p.sender).toMatchObject({ login: "ada", id: "u1", type: "User" }); | |
| 340 | + expect(p.organization).toMatchObject({ login: "acme" }); | |
| 341 | + expect(p.ref).toBe("refs/heads/main"); | |
| 342 | + }); | |
| 343 | + | |
| 344 | + it("completes pull_request with head.repo and keeps trigger fields", () => { | |
| 345 | + const p = completePayload(site, { action: "opened", number: 3, pull_request: { number: 3, title: "T", head: { ref: "feat", sha: "abc" }, base: { ref: "main", sha: "def" }, user: { login: "bob" } } }, { login: "bob", kind: "agent", id: "a1" }); | |
| 346 | + const pr = p.pull_request as Record<string, any>; | |
| 347 | + expect(pr.head).toMatchObject({ ref: "feat", sha: "abc", label: "acme:feat" }); | |
| 348 | + expect(pr.head.repo.full_name).toBe("acme/app"); | |
| 349 | + expect(pr.base.repo.full_name).toBe("acme/app"); | |
| 350 | + expect(pr).toMatchObject({ number: 3, title: "T", state: "open", html_url: "https://d.example/acme/app/pull/3" }); | |
| 351 | + expect(pr.user.login).toBe("bob"); | |
| 352 | + expect((p.sender as Record<string, unknown>).type).toBe("Bot"); | |
| 353 | + }); | |
| 354 | + | |
| 355 | + it("builds issue and release objects", () => { | |
| 356 | + const issue = issuePayload(site, { number: 5, kind: "issue", title: "Bug", body: "b", state: "open", stateReason: null, author: { id: "u", login: "ada", kind: "user" }, assignees: ["bob"], labels: ["bug"], createdAt: "t", updatedAt: "t", closedAt: null, comments: 2 }); | |
| 357 | + expect(issue).toMatchObject({ number: 5, title: "Bug", labels: [{ name: "bug" }], comments: 2, html_url: "https://d.example/acme/app/issues/5" }); | |
| 358 | + expect(issue.pull_request).toBeUndefined(); | |
| 359 | + const rel = releasePayload(site, { id: 1, tagName: "v1.0.0", targetCommitish: "main", name: "v1", body: null, draft: false, prerelease: false, createdAt: "t", publishedAt: "t", author: { login: "ada" } }); | |
| 360 | + expect(rel).toMatchObject({ tag_name: "v1.0.0", html_url: "https://d.example/acme/app/releases/tag/v1.0.0", upload_url: "https://d.example/api/v3/repos/acme/app/releases/1/assets{?name,label}" }); | |
| 361 | + }); | |
| 362 | +}); | |
| 363 | + | |
| 364 | +/** The Durable Object's SqlStorage over node:sqlite: exec(query, ...bindings) with toArray() and one(). */ | |
| 365 | +function sqlStorage() { | |
| 366 | + const db = new DatabaseSync(":memory:"); | |
| 367 | + return { | |
| 368 | + exec(query: string, ...args: unknown[]) { | |
| 369 | + const params = args.map((a) => (a === undefined ? null : a)) as Array<string | number | null>; | |
| 370 | + let rows: unknown[] = []; | |
| 371 | + if (!params.length && query.trim().replace(/;\s*$/, "").includes(";")) db.exec(query); | |
| 372 | + else rows = db.prepare(query).all(...params); | |
| 373 | + return { | |
| 374 | + toArray: () => rows, | |
| 375 | + one: () => { | |
| 376 | + if (rows.length !== 1) throw new Error(`expected one row, got ${rows.length}`); | |
| 377 | + return rows[0]; | |
| 378 | + }, | |
| 379 | + }; | |
| 380 | + }, | |
| 381 | + }; | |
| 382 | +} | |
| 383 | + | |
| 384 | +/** A Runtime over a fresh repository schema, an in-memory bucket, and jobs running on given refs. */ | |
| 385 | +function runtimeFixture() { | |
| 386 | + const sql = sqlStorage(); | |
| 387 | + sql.exec(REPO_SCHEMA); | |
| 388 | + migrateRepoSchema(sql as unknown as SqlStorage); | |
| 389 | + const m = memoryBucket(); | |
| 390 | + const repo = { sql, meta: { repoId: "r", defaultBranch: "main" } }; | |
| 391 | + const rt = new Runtime(repo as unknown as RepoDO, { BUCKET: m.bucket } as unknown as Env); | |
| 392 | + let n = 0; | |
| 393 | + /** A running job of a run on `ref`; `payload` is the run's event payload. */ | |
| 394 | + const job = (ref: string, payload: Record<string, unknown> = {}) => { | |
| 395 | + n++; | |
| 396 | + const t = new Date().toISOString(); | |
| 397 | + sql.exec( | |
| 398 | + "INSERT INTO runs (id, run_number, workflow_path, workflow_name, event, ref, sha, actor, status, created_at, updated_at, event_payload) VALUES (?, ?, 'w.yml', 'w', 'push', ?, 'abc', '{}', 'in_progress', ?, ?, ?)", | |
| 399 | + `run${n}`, | |
| 400 | + n, | |
| 401 | + ref, | |
| 402 | + t, | |
| 403 | + t, | |
| 404 | + JSON.stringify(payload), | |
| 405 | + ); | |
| 406 | + sql.exec("INSERT INTO jobs (id, run_id, job_key, name, status) VALUES (?, ?, 'j', 'j', 'in_progress')", `job${n}`, `run${n}`); | |
| 407 | + return `job${n}`; | |
| 408 | + }; | |
| 409 | + const version = "ab".repeat(32); | |
| 410 | + /** Saves `key` from `jobId` (reserve, then commit). */ | |
| 411 | + const save = async (jobId: string, key: string) => { | |
| 412 | + const r = await rt.cacheReserve(jobId, key, version); | |
| 413 | + expect(r).not.toBeNull(); | |
| 414 | + await rt.cacheCommit(jobId, r!.id, 10, "etag"); | |
| 415 | + }; | |
| 416 | + const restore = (jobId: string, ...keys: string[]) => rt.cacheLookup(jobId, keys, version); | |
| 417 | + return { sql, rt, m, job, save, restore }; | |
| 418 | +} | |
| 419 | + | |
| 420 | +describe("cache scopes (as on GitHub)", () => { | |
| 421 | + it("restores from the job's own ref, then the default branch; never another branch's", async () => { | |
| 422 | + const f = runtimeFixture(); | |
| 423 | + const main = f.job("refs/heads/main"); | |
| 424 | + await f.save(main, "deps-main"); | |
| 425 | + const a = f.job("refs/heads/a"); | |
| 426 | + await f.save(a, "deps-a"); | |
| 427 | + // a restores its own entry first, else main's | |
| 428 | + expect(f.restore(a, "deps-a")?.scope).toBe("refs/heads/a"); | |
| 429 | + expect(f.restore(a, "nothing", "deps-")?.key).toBe("deps-a"); | |
| 430 | + expect(f.restore(a, "deps-main")?.scope).toBe("refs/heads/main"); | |
| 431 | + // b sees main's, not a's newer one, even by prefix | |
| 432 | + const b = f.job("refs/heads/b"); | |
| 433 | + expect(f.restore(b, "deps-a")).toBeNull(); | |
| 434 | + expect(f.restore(b, "deps-")?.key).toBe("deps-main"); | |
| 435 | + // the default branch never sees a branch's caches | |
| 436 | + const main2 = f.job("refs/heads/main"); | |
| 437 | + expect(f.restore(main2, "deps-a")).toBeNull(); | |
| 438 | + expect(f.restore(main2, "deps-")?.key).toBe("deps-main"); | |
| 439 | + // saves go to the job's own ref: b may save a key that a already has | |
| 440 | + await f.save(b, "deps-a"); | |
| 441 | + expect(f.restore(b, "deps-a")?.scope).toBe("refs/heads/b"); | |
| 442 | + }); | |
| 443 | + | |
| 444 | + it("gives pull requests their base branch's caches but not other pull requests'", async () => { | |
| 445 | + const f = runtimeFixture(); | |
| 446 | + await f.save(f.job("refs/heads/main"), "k-main"); | |
| 447 | + await f.save(f.job("refs/heads/release"), "k-release"); | |
| 448 | + const pr1 = f.job("refs/pull/1/merge", { pull_request: { number: 1, base: { ref: "release" } } }); | |
| 449 | + expect(f.restore(pr1, "k-")?.key).toBe("k-release"); | |
| 450 | + await f.save(pr1, "k-pr1"); | |
| 451 | + expect(f.restore(pr1, "k-")?.key).toBe("k-pr1"); | |
| 452 | + const pr2 = f.job("refs/pull/2/merge", { pull_request: { number: 2, base: { ref: "main" } } }); | |
| 453 | + expect(f.restore(pr2, "k-pr1")).toBeNull(); | |
| 454 | + expect(f.restore(pr2, "k-release")).toBeNull(); | |
| 455 | + expect(f.restore(pr2, "k-")?.key).toBe("k-main"); | |
| 456 | + // nor the base branch or the default branch theirs | |
| 457 | + expect(f.restore(f.job("refs/heads/release"), "k-pr1")).toBeNull(); | |
| 458 | + expect(f.restore(f.job("refs/heads/main"), "k-pr")).toBeNull(); | |
| 459 | + // a merge group reads its target branch's caches | |
| 460 | + const group = f.job("refs/heads/gh-readonly-queue/release/pr-1-abc", { merge_group: { base_ref: "refs/heads/release" } }); | |
| 461 | + expect(f.restore(group, "k-")?.key).toBe("k-release"); | |
| 462 | + }); | |
| 463 | +}); | |
| 464 | + | |
| 465 | +describe("artifact retention", () => { | |
| 466 | + const DAY = 86400_000; | |
| 467 | + async function upload(f: ReturnType<typeof runtimeFixture>, jobId: string, name: string, expiresAt?: string) { | |
| 468 | + const { id } = await f.rt.artifactCreate(jobId, name, undefined, expiresAt); | |
| 469 | + await f.m.bucket.put(finalKey({ repo: "r", kind: "artifact", id }), new Uint8Array(100)); | |
| 470 | + await f.rt.stage("artifact", id, 100); | |
| 471 | + return f.rt.artifactFinalize(jobId, name, 100, undefined, "etag"); | |
| 472 | + } | |
| 473 | + const days = (iso: string) => Math.round((Date.parse(iso) - Date.now()) / DAY); | |
| 474 | + | |
| 475 | + it("honours retention-days, clamped to 1..90 days", async () => { | |
| 476 | + const f = runtimeFixture(); | |
| 477 | + const j = f.job("refs/heads/main"); | |
| 478 | + expect(days((await upload(f, j, "five", new Date(Date.now() + 5 * DAY).toISOString())).expiresAt)).toBe(5); | |
| 479 | + expect(days((await upload(f, j, "default")).expiresAt)).toBe(90); | |
| 480 | + expect(days((await upload(f, j, "too-long", new Date(Date.now() + 400 * DAY).toISOString())).expiresAt)).toBe(90); | |
| 481 | + expect(days((await upload(f, j, "too-short", new Date(Date.now() - DAY).toISOString())).expiresAt)).toBe(1); | |
| 482 | + expect(days((await upload(f, j, "garbage", "next tuesday")).expiresAt)).toBe(90); | |
| 483 | + }); | |
| 484 | + | |
| 485 | + it("deletes an expired artifact's content and lists it as expired", async () => { | |
| 486 | + const f = runtimeFixture(); | |
| 487 | + const j = f.job("refs/heads/main"); | |
| 488 | + const a = await upload(f, j, "old", new Date(Date.now() + 2 * DAY).toISOString()); | |
| 489 | + const keep = await upload(f, j, "new", new Date(Date.now() + 3 * DAY).toISOString()); | |
| 490 | + expect(a.expired).toBe(false); | |
| 491 | + f.sql.exec("UPDATE rt_artifacts SET expires_at = ? WHERE id = ?", new Date(Date.now() - 1000).toISOString(), a.id); | |
| 492 | + // Already shown as expired (and not handed to the toolkit) before the sweep runs. | |
| 493 | + expect(f.rt.artifactGet(a.id)).toBeNull(); | |
| 494 | + expect(f.rt.artifactGet(a.id, { includeExpired: true })?.expired).toBe(true); | |
| 495 | + // A job ending sweeps: the content goes, the listing stays. | |
| 496 | + await f.rt.sweepJob(j); | |
| 497 | + expect(f.m.objects.has(finalKey({ repo: "r", kind: "artifact", id: a.id }))).toBe(false); | |
| 498 | + expect(f.m.objects.has(finalKey({ repo: "r", kind: "artifact", id: keep.id }))).toBe(true); | |
| 499 | + expect(f.rt.runArtifacts("run1").map((x) => x.name)).toEqual(["new"]); | |
| 500 | + expect(f.rt.runArtifacts("run1", { includeExpired: true }).map((x) => [x.name, x.expired])).toEqual([ | |
| 501 | + ["new", false], | |
| 502 | + ["old", true], | |
| 503 | + ]); | |
| 504 | + const listed = f.rt.repoArtifacts({}); | |
| 505 | + expect(listed.artifacts.find((x) => x.id === a.id)?.expired).toBe(true); | |
| 506 | + // Expired artifacts no longer count against the quota. | |
| 507 | + expect(f.sql.exec("SELECT status, staged FROM rt_artifacts WHERE id = ?", a.id).one()).toEqual({ status: "expired", staged: 0 }); | |
| 508 | + // The name can be used again, and 90 days on the listing goes too. | |
| 509 | + const again = f.job("refs/heads/main"); | |
| 510 | + f.sql.exec("UPDATE jobs SET run_id = 'run1' WHERE id = ?", again); | |
| 511 | + expect((await upload(f, again, "old")).expired).toBe(false); | |
| 512 | + f.sql.exec("UPDATE rt_artifacts SET expires_at = ? WHERE id = ?", new Date(Date.now() - 91 * DAY).toISOString(), keep.id); | |
| 513 | + await f.rt.expireArtifacts(); | |
| 514 | + await f.rt.expireArtifacts(); | |
| 515 | + expect(f.rt.artifactGet(keep.id, { includeExpired: true })).toBeNull(); | |
| 516 | + }); | |
| 517 | +}); |
addedapps/diffinery/test/unit/job-graph.test.ts+139−0View file
| @@ -1,0 +1,139 @@ | ||
| 1 | +import { describe, expect, it } from "vitest"; | |
| 2 | +import { evaluateIf } from "@diffinery/workflow-yaml"; | |
| 3 | +import { ancestors, combineResults, graphFromLevels, jobStatus, runGraph, runnerLabels } from "../../src/repo/job-graph.js"; | |
| 4 | + | |
| 5 | +describe("ancestors", () => { | |
| 6 | + it("follows needs transitively, once per job", () => { | |
| 7 | + const needs = new Map([ | |
| 8 | + ["deploy", ["test", "lint"]], | |
| 9 | + ["test", ["build"]], | |
| 10 | + ["lint", ["build"]], | |
| 11 | + ["build", []], | |
| 12 | + ]); | |
| 13 | + expect(ancestors(needs, ["test", "lint"]).sort()).toEqual(["build", "lint", "test"]); | |
| 14 | + expect(ancestors(needs, [])).toEqual([]); | |
| 15 | + }); | |
| 16 | +}); | |
| 17 | + | |
| 18 | +describe("jobStatus", () => { | |
| 19 | + const cond = (expr: string | undefined, results: Parameters<typeof jobStatus>[0], cancelled = false) => evaluateIf(expr, {}, jobStatus(results, cancelled)); | |
| 20 | + | |
| 21 | + it("skips a job whose ancestor was skipped, even when its own need succeeded", () => { | |
| 22 | + // a (skipped) <- b (if: always(), succeeded) <- c: c is skipped on GitHub. | |
| 23 | + expect(cond(undefined, ["skipped", "success"])).toBe(false); | |
| 24 | + expect(cond("success()", ["skipped"])).toBe(false); | |
| 25 | + expect(cond("failure()", ["skipped"])).toBe(false); | |
| 26 | + expect(cond("always()", ["skipped"])).toBe(true); | |
| 27 | + // The usual workaround runs it. | |
| 28 | + expect(cond("!cancelled() && !failure()", ["skipped", "success"])).toBe(true); | |
| 29 | + }); | |
| 30 | + | |
| 31 | + it("runs on success, and failure() sees a failed ancestor anywhere up the chain", () => { | |
| 32 | + expect(cond(undefined, ["success", "success"])).toBe(true); | |
| 33 | + expect(cond(undefined, [])).toBe(true); | |
| 34 | + expect(cond(undefined, ["failure", "success"])).toBe(false); | |
| 35 | + expect(cond("failure()", ["failure", "success"])).toBe(true); | |
| 36 | + expect(cond("failure()", ["cancelled"])).toBe(false); | |
| 37 | + }); | |
| 38 | + | |
| 39 | + it("knows about cancelled runs", () => { | |
| 40 | + expect(cond(undefined, ["success"], true)).toBe(false); | |
| 41 | + expect(cond("cancelled()", ["success"], true)).toBe(true); | |
| 42 | + expect(cond("always()", ["cancelled"], true)).toBe(true); | |
| 43 | + }); | |
| 44 | +}); | |
| 45 | + | |
| 46 | +describe("combineResults", () => { | |
| 47 | + it("is skipped for a job that never ran or whose legs were all skipped", () => { | |
| 48 | + expect(combineResults([])).toBe("skipped"); | |
| 49 | + expect(combineResults([{ conclusion: "skipped" }, { conclusion: "skipped" }])).toBe("skipped"); | |
| 50 | + }); | |
| 51 | + | |
| 52 | + it("lets failure win over cancellation, and counts a timeout as a failure", () => { | |
| 53 | + expect(combineResults([{ conclusion: "success" }, { conclusion: "cancelled" }])).toBe("cancelled"); | |
| 54 | + expect(combineResults([{ conclusion: "cancelled" }, { conclusion: "failure" }])).toBe("failure"); | |
| 55 | + expect(combineResults([{ conclusion: "timed_out" }])).toBe("failure"); | |
| 56 | + }); | |
| 57 | + | |
| 58 | + it("treats a failed leg with continue-on-error as a success", () => { | |
| 59 | + expect(combineResults([{ conclusion: "success" }, { conclusion: "failure", continueOnError: true }])).toBe("success"); | |
| 60 | + expect(combineResults([{ conclusion: "failure", continueOnError: true }, { conclusion: "failure" }])).toBe("failure"); | |
| 61 | + }); | |
| 62 | +}); | |
| 63 | + | |
| 64 | +describe("runnerLabels", () => { | |
| 65 | + const ctx = { matrix: { os: "ubuntu-24.04", labels: ["self-hosted", "gpu"] }, inputs: { runner: "linux" } }; | |
| 66 | + | |
| 67 | + it("evaluates matrix and other contexts per leg", () => { | |
| 68 | + expect(runnerLabels(["${{ matrix.os }}"], ctx)).toEqual(["ubuntu-24.04"]); | |
| 69 | + expect(runnerLabels(["self-hosted", "${{ inputs.runner }}"], ctx)).toEqual(["self-hosted", "linux"]); | |
| 70 | + }); | |
| 71 | + | |
| 72 | + it("flattens an expression that yields a list, and drops empty labels", () => { | |
| 73 | + expect(runnerLabels(["${{ matrix.labels }}"], ctx)).toEqual(["self-hosted", "gpu"]); | |
| 74 | + expect(runnerLabels(["${{ matrix.missing }}", "linux"], ctx)).toEqual(["linux"]); | |
| 75 | + }); | |
| 76 | +}); | |
| 77 | + | |
| 78 | +describe("runGraph", () => { | |
| 79 | + /** A controllable job: resolves when the test finishes it. */ | |
| 80 | + function harness() { | |
| 81 | + const started: string[] = []; | |
| 82 | + const finish = new Map<string, () => void>(); | |
| 83 | + const start = (key: string) => | |
| 84 | + new Promise<void>((resolve) => { | |
| 85 | + started.push(key); | |
| 86 | + finish.set(key, resolve); | |
| 87 | + }); | |
| 88 | + const tick = () => new Promise((r) => setTimeout(r, 0)); | |
| 89 | + const finishJob = async (key: string) => { | |
| 90 | + finish.get(key)!(); | |
| 91 | + await tick(); | |
| 92 | + }; | |
| 93 | + return { started, finish: finishJob, start, tick }; | |
| 94 | + } | |
| 95 | + | |
| 96 | + it("starts a job as soon as its own needs finish, not when its whole level does", async () => { | |
| 97 | + const h = harness(); | |
| 98 | + // build-a and build-b have no needs; test-a only needs build-a; deploy needs both tests. | |
| 99 | + const done = runGraph( | |
| 100 | + [ | |
| 101 | + { key: "build-a", needs: [] }, | |
| 102 | + { key: "build-b", needs: [] }, | |
| 103 | + { key: "test-a", needs: ["build-a"] }, | |
| 104 | + { key: "test-b", needs: ["build-b"] }, | |
| 105 | + { key: "deploy", needs: ["test-a", "test-b"] }, | |
| 106 | + ], | |
| 107 | + h.start, | |
| 108 | + ); | |
| 109 | + await h.tick(); | |
| 110 | + expect(h.started).toEqual(["build-a", "build-b"]); | |
| 111 | + await h.finish("build-a"); | |
| 112 | + // build-b still runs: a level-by-level scheduler would hold test-a back here. | |
| 113 | + expect(h.started).toEqual(["build-a", "build-b", "test-a"]); | |
| 114 | + await h.finish("test-a"); | |
| 115 | + expect(h.started).not.toContain("deploy"); | |
| 116 | + await h.finish("build-b"); | |
| 117 | + expect(h.started).toEqual(["build-a", "build-b", "test-a", "test-b"]); | |
| 118 | + await h.finish("test-b"); | |
| 119 | + expect(h.started.at(-1)).toBe("deploy"); | |
| 120 | + await h.finish("deploy"); | |
| 121 | + await done; | |
| 122 | + expect(h.started).toHaveLength(5); | |
| 123 | + }); | |
| 124 | + | |
| 125 | + it("ignores needs outside the graph", async () => { | |
| 126 | + const started: string[] = []; | |
| 127 | + await runGraph([{ key: "a", needs: ["gone"] }], async (k) => void started.push(k)); | |
| 128 | + expect(started).toEqual(["a"]); | |
| 129 | + }); | |
| 130 | + | |
| 131 | + it("turns recorded levels into a graph where each level waits for the previous one", () => { | |
| 132 | + expect(graphFromLevels([["a", "b"], ["c"], ["d"]])).toEqual([ | |
| 133 | + { key: "a", needs: [] }, | |
| 134 | + { key: "b", needs: [] }, | |
| 135 | + { key: "c", needs: ["a", "b"] }, | |
| 136 | + { key: "d", needs: ["c"] }, | |
| 137 | + ]); | |
| 138 | + }); | |
| 139 | +}); |
addedapps/diffinery/test/unit/job-permissions.test.ts+161−0View file
| @@ -1,0 +1,161 @@ | ||
| 1 | +import { describe, expect, it } from "vitest"; | |
| 2 | +import { parseWorkflow, type Workflow } from "@diffinery/workflow-yaml"; | |
| 3 | +import { | |
| 4 | + PERMISSIVE_DEFAULT, | |
| 5 | + READ_ALL, | |
| 6 | + RESTRICTED_DEFAULT, | |
| 7 | + denied, | |
| 8 | + permissionsForJob, | |
| 9 | + readOnly, | |
| 10 | + requestPermission, | |
| 11 | + resolvePermissions, | |
| 12 | + roleForPermissions, | |
| 13 | + type JobPermissions, | |
| 14 | +} from "../../src/lib/job-permissions.js"; | |
| 15 | +import { expandReusable, type Loader } from "../../src/repo/reusable.js"; | |
| 16 | + | |
| 17 | +const wf = (yaml: string): Workflow => { | |
| 18 | + const { workflow, errors } = parseWorkflow(yaml, ".github/workflows/x.yml"); | |
| 19 | + if (!workflow) throw new Error(errors.map((e) => e.message).join("; ")); | |
| 20 | + return workflow; | |
| 21 | +}; | |
| 22 | +const job = (perms?: string) => ` runs-on: ubuntu-latest\n${perms ? ` permissions: ${perms}\n` : ""} steps: [{ run: echo }]\n`; | |
| 23 | +const expand = async (root: Workflow, lib?: Workflow) => { | |
| 24 | + const load: Loader = async (_uses, from) => ({ workflow: lib!, source: from }); | |
| 25 | + return expandReusable(root, { repoId: null, sha: "abc" }, load); | |
| 26 | +}; | |
| 27 | +const writes = (p: JobPermissions) => Object.entries(p).filter(([, v]) => v === "write").map(([k]) => k).sort(); | |
| 28 | + | |
| 29 | +describe("permissions: as written", () => { | |
| 30 | + it("resolves read-all, write-all, {} and mappings like GitHub", () => { | |
| 31 | + expect(resolvePermissions("read-all")).toEqual(READ_ALL); | |
| 32 | + expect(resolvePermissions("write-all")["id-token"]).toBe("write"); | |
| 33 | + expect(resolvePermissions("write-all").models).toBe("read"); | |
| 34 | + expect(Object.values(resolvePermissions({})).every((v) => v === "none")).toBe(true); | |
| 35 | + const p = resolvePermissions({ contents: "read", issues: "write", "id-token": "read" }); | |
| 36 | + expect(p).toMatchObject({ contents: "read", issues: "write", "id-token": "none", statuses: "none" }); | |
| 37 | + expect(PERMISSIVE_DEFAULT).toMatchObject({ contents: "write", issues: "write", packages: "write", "id-token": "none" }); | |
| 38 | + expect(roleForPermissions(resolvePermissions({ issues: "write" }))).toBe("write"); | |
| 39 | + expect(roleForPermissions(resolvePermissions({ "id-token": "write", contents: "read" }))).toBe("read"); | |
| 40 | + }); | |
| 41 | +}); | |
| 42 | + | |
| 43 | +describe("effective permissions of a job", () => { | |
| 44 | + it("uses the job's, else the workflow's, else the permissive default", async () => { | |
| 45 | + const exp = await expand( | |
| 46 | + wf(`on: push | |
| 47 | +permissions: { contents: read, issues: write } | |
| 48 | +jobs: | |
| 49 | + inherits: | |
| 50 | +${job()} own: | |
| 51 | +${job("{ statuses: write }")}`), | |
| 52 | + ); | |
| 53 | + expect(writes(permissionsForJob(exp, "inherits"))).toEqual(["issues"]); | |
| 54 | + expect(permissionsForJob(exp, "inherits").contents).toBe("read"); | |
| 55 | + // Job level replaces the workflow level entirely. | |
| 56 | + expect(writes(permissionsForJob(exp, "own"))).toEqual(["statuses"]); | |
| 57 | + expect(permissionsForJob(exp, "own").contents).toBe("none"); | |
| 58 | + const open = await expand(wf(`on: push\njobs:\n a:\n${job()}`)); | |
| 59 | + expect(permissionsForJob(open, "a")).toEqual(PERMISSIVE_DEFAULT); | |
| 60 | + }); | |
| 61 | + | |
| 62 | + it("defaults to the repository's setting: read-only contents and packages, or everything", async () => { | |
| 63 | + const open = await expand(wf(`on: push\njobs:\n a:\n${job()}`)); | |
| 64 | + expect(permissionsForJob(open, "a", { repoDefault: "read" })).toEqual(RESTRICTED_DEFAULT); | |
| 65 | + expect(writes(RESTRICTED_DEFAULT)).toEqual([]); | |
| 66 | + expect(RESTRICTED_DEFAULT).toMatchObject({ contents: "read", packages: "read", issues: "none", "pull-requests": "none" }); | |
| 67 | + expect(permissionsForJob(open, "a", { repoDefault: "write" })).toEqual(PERMISSIVE_DEFAULT); | |
| 68 | + // what a workflow asks for is not limited by the default | |
| 69 | + const asks = await expand(wf(`on: push\npermissions: { issues: write }\njobs:\n a:\n${job()}`)); | |
| 70 | + expect(writes(permissionsForJob(asks, "a", { repoDefault: "read" }))).toEqual(["issues"]); | |
| 71 | + }); | |
| 72 | + | |
| 73 | + it("never gives a fork's pull request write", async () => { | |
| 74 | + const exp = await expand(wf(`on: pull_request\npermissions: write-all\njobs:\n a:\n${job()}`)); | |
| 75 | + expect(writes(permissionsForJob(exp, "a", { untrusted: true }))).toEqual([]); | |
| 76 | + expect(permissionsForJob(exp, "a", { untrusted: true }).contents).toBe("read"); | |
| 77 | + }); | |
| 78 | + | |
| 79 | + it("caps a reusable workflow by the calling job", async () => { | |
| 80 | + const lib = wf(`on: workflow_call | |
| 81 | +jobs: | |
| 82 | + plain: | |
| 83 | +${job()} greedy: | |
| 84 | +${job("write-all")}`); | |
| 85 | + const exp = await expand( | |
| 86 | + wf(`on: push | |
| 87 | +permissions: read-all | |
| 88 | +jobs: | |
| 89 | + call: | |
| 90 | + permissions: { contents: read, issues: write } | |
| 91 | + uses: ./.github/workflows/lib.yml | |
| 92 | +`), | |
| 93 | + lib, | |
| 94 | + ); | |
| 95 | + expect(writes(permissionsForJob(exp, "call/plain"))).toEqual(["issues"]); | |
| 96 | + expect(writes(permissionsForJob(exp, "call/greedy"))).toEqual(["issues"]); | |
| 97 | + expect(permissionsForJob(exp, "call/greedy")["id-token"]).toBe("none"); | |
| 98 | + }); | |
| 99 | +}); | |
| 100 | + | |
| 101 | +describe("what each request needs", () => { | |
| 102 | + const p = resolvePermissions({ contents: "read", issues: "write" }); | |
| 103 | + const check = (method: string, path: string, min: "read" | "write" = "read") => denied(p, requestPermission(method, path, min)); | |
| 104 | + | |
| 105 | + it("maps endpoints to permission scopes", () => { | |
| 106 | + expect(check("POST", "/api/v3/repos/o/r/issues")).toBeNull(); | |
| 107 | + expect(check("POST", "/api/v3/repos/o/r/issues/3/comments")).toBeNull(); | |
| 108 | + expect(check("POST", "/api/v3/repos/o/r/labels")).toBeNull(); | |
| 109 | + expect(check("GET", "/api/v3/repos/o/r/contents/README.md")).toBeNull(); | |
| 110 | + expect(check("GET", "/api/v3/repos/o/r")).toBeNull(); | |
| 111 | + expect(check("PUT", "/api/v3/repos/o/r/contents/x")).toMatch(/contents: write/); | |
| 112 | + expect(check("POST", "/api/v3/repos/o/r/statuses/abc")).toMatch(/statuses: write/); | |
| 113 | + expect(check("POST", "/api/v3/repos/o/r/check-runs")).toMatch(/checks: write/); | |
| 114 | + expect(check("POST", "/api/v3/repos/o/r/pulls")).toMatch(/pull-requests: write/); | |
| 115 | + expect(check("PUT", "/api/v3/repos/o/r/pulls/1/merge")).toMatch(/contents: write/); | |
| 116 | + expect(check("GET", "/api/v3/repos/o/r/actions/runs")).toMatch(/actions: read/); | |
| 117 | + expect(check("POST", "/api/v3/repos/o/r/releases")).toMatch(/contents: write/); | |
| 118 | + expect(check("POST", "/api/uploads/repos/o/r/releases/1/assets")).toMatch(/contents: write/); | |
| 119 | + expect(check("POST", "/api/v3/repos/o/r/commits/abc/comments")).toMatch(/contents: write/); | |
| 120 | + expect(check("PATCH", "/api/v3/repos/o/r/comments/7")).toMatch(/contents: write/); | |
| 121 | + expect(check("PATCH", "/api/v3/repos/o/r/issues/comments/7")).toBeNull(); | |
| 122 | + // git: fetches are POSTs that read; pushes need contents: write | |
| 123 | + expect(check("POST", "/o/r.git/git-upload-pack")).toBeNull(); | |
| 124 | + expect(check("GET", "/o/r.git/info/refs", "write")).toMatch(/contents: write/); | |
| 125 | + expect(check("POST", "/o/r.git/info/lfs/objects/batch")).toBeNull(); | |
| 126 | + // web pages: the code, a commit, the editor, and pull requests by their web path | |
| 127 | + expect(check("GET", "/o/r")).toBeNull(); | |
| 128 | + expect(requestPermission("GET", "/o/r", "read")).toEqual({ scopes: ["contents"], access: "read" }); | |
| 129 | + expect(requestPermission("GET", "/o/r/commit/abc", "read")).toEqual({ scopes: ["contents"], access: "read" }); | |
| 130 | + expect(check("POST", "/o/r/edit/main/README.md")).toMatch(/contents: write/); | |
| 131 | + expect(check("POST", "/o/r/new/main")).toMatch(/contents: write/); | |
| 132 | + expect(check("POST", "/o/r/pull/1/merge")).toMatch(/pull-requests: write/); | |
| 133 | + expect(requestPermission("GET", "/o/r/pull/1", "read")).toEqual({ scopes: ["pull-requests"], access: "read" }); | |
| 134 | + }); | |
| 135 | + | |
| 136 | + it("lets anyone's reads of a public repository through, never its writes", () => { | |
| 137 | + expect(readOnly(requestPermission("GET", "/api/v3/repos/o/r", "read"))).toBe(true); | |
| 138 | + expect(readOnly(requestPermission("GET", "/api/v3/repos/o/r/contents/x", "read"))).toBe(true); | |
| 139 | + expect(readOnly(requestPermission("POST", "/o/r.git/git-upload-pack", "read"))).toBe(true); | |
| 140 | + expect(readOnly(requestPermission("PUT", "/api/v3/repos/o/r/contents/x", "write"))).toBe(false); | |
| 141 | + expect(readOnly(requestPermission("GET", "/o/r.git/info/refs", "write"))).toBe(false); | |
| 142 | + expect(readOnly(requestPermission("GET", "/api/v3/repos/o/r/actions/secrets", "read"))).toBe(false); | |
| 143 | + // contents: none still blocks the code (of a private repository; loadRepo lets public reads through) | |
| 144 | + const none = resolvePermissions({ contents: "none" }); | |
| 145 | + expect(denied(none, requestPermission("GET", "/api/v3/repos/o/r/contents/x", "read"))).toMatch(/contents: read/); | |
| 146 | + }); | |
| 147 | + | |
| 148 | + it("keeps repository administration away from job tokens", () => { | |
| 149 | + const all = resolvePermissions("write-all"); | |
| 150 | + const admin = (method: string, path: string) => denied(all, requestPermission(method, path, "read")); | |
| 151 | + expect(admin("PATCH", "/api/v3/repos/o/r")).toMatch(/not available|cannot/); | |
| 152 | + expect(admin("PUT", "/api/v3/repos/o/r/actions/secrets/X")).toMatch(/not available/); | |
| 153 | + expect(admin("GET", "/api/v3/repos/o/r/actions/secrets")).toMatch(/not available/); | |
| 154 | + expect(admin("PUT", "/api/v3/repos/o/r/branches/main/protection")).toMatch(/not available/); | |
| 155 | + expect(admin("POST", "/api/v3/repos/o/r/hooks")).toMatch(/not available/); | |
| 156 | + expect(admin("POST", "/api/v3/repos/o/r/actions/runners/claim")).toMatch(/not available/); | |
| 157 | + expect(admin("POST", "/mcp")).toMatch(/not available/); | |
| 158 | + // a file or branch that happens to be called "settings" or "secrets" is still just contents | |
| 159 | + expect(admin("GET", "/api/v3/repos/o/r/contents/settings/secrets.md")).toBeNull(); | |
| 160 | + }); | |
| 161 | +}); |
addedapps/diffinery/test/unit/webhook-payload.test.ts+36−0View file
| @@ -1,0 +1,36 @@ | ||
| 1 | +import type { DiffineryEvent } from "@diffinery/shared"; | |
| 2 | +import { describe, expect, it } from "vitest"; | |
| 3 | +import type { Repo } from "../../src/lib/directory.js"; | |
| 4 | +import { webhookPayload } from "../../src/lib/webhook-payload.js"; | |
| 5 | + | |
| 6 | +const BASE = "https://diffinery.example"; | |
| 7 | +const row = (visibility: Repo["visibility"]): Repo => ({ | |
| 8 | + id: "r1", | |
| 9 | + owner_id: "u1", | |
| 10 | + owner_login: "octo", | |
| 11 | + owner_kind: "user", | |
| 12 | + name: "app", | |
| 13 | + description: "", | |
| 14 | + visibility, | |
| 15 | + default_branch: "main", | |
| 16 | + archived: 0, | |
| 17 | + created_at: "2026-01-01T00:00:00.000Z", | |
| 18 | + updated_at: "2026-01-01T00:00:00.000Z", | |
| 19 | + pushed_at: null, | |
| 20 | +}); | |
| 21 | +const push: Extract<DiffineryEvent, { type: "push" }> = { type: "push", repoId: "r1", owner: "octo", repo: "app", ref: "refs/heads/main", before: "0".repeat(40), after: "a".repeat(40), forced: false, created: false, deleted: false, actor: { id: "u1", login: "octo", kind: "user" }, at: "2026-01-01T00:00:00.000Z" }; | |
| 22 | + | |
| 23 | +describe("webhookPayload", () => { | |
| 24 | + it("reports a public repository as public, like GitHub", () => { | |
| 25 | + const body = webhookPayload(BASE, push, row("public")); | |
| 26 | + expect(body.repository).toMatchObject({ full_name: "octo/app", private: false, visibility: "public", default_branch: "main", owner: { login: "octo" } }); | |
| 27 | + }); | |
| 28 | + | |
| 29 | + it("reports a private repository as private", () => { | |
| 30 | + expect(webhookPayload(BASE, push, row("private")).repository).toMatchObject({ private: true, visibility: "private" }); | |
| 31 | + }); | |
| 32 | + | |
| 33 | + it("falls back to a minimal private repository when the directory row is gone", () => { | |
| 34 | + expect(webhookPayload(BASE, push, null).repository).toMatchObject({ full_name: "octo/app", private: true, visibility: "private" }); | |
| 35 | + }); | |
| 36 | +}); |
modifiedcontainers/runner/Dockerfile+14−1View file
| @@ -14,13 +14,16 @@ | ||
| 14 | 14 | RUNNER_TOOL_CACHE=/opt/hostedtoolcache \ |
| 15 | 15 | AGENT_TOOLSDIRECTORY=/opt/hostedtoolcache \ |
| 16 | 16 | COREPACK_ENABLE_DOWNLOAD_PROMPT=0 \ |
| 17 | + # What hosted ubuntu-24.04 runners report; setup-* actions key their caches on it. | |
| 18 | + ImageOS=ubuntu24 \ | |
| 19 | + ImageVersion=20250101.1 \ | |
| 17 | 20 | # Ubuntu 24.04 marks the system Python as externally managed (PEP 668); |
| 18 | 21 | # CI scripts commonly `pip install` directly, so allow it like hosted runners do. |
| 19 | 22 | PIP_BREAK_SYSTEM_PACKAGES=1 |
| 20 | 23 | |
| 21 | 24 | RUN apt-get update \ |
| 22 | 25 | && apt-get install -y --no-install-recommends \ |
| 23 | - ca-certificates curl git git-lfs jq zip unzip xz-utils tar gzip \ | |
| 26 | + ca-certificates curl git git-lfs jq zip unzip xz-utils tar gzip zstd \ | |
| 24 | 27 | build-essential pkg-config openssh-client gnupg \ |
| 25 | 28 | python3 python3-pip python3-venv python-is-python3 \ |
| 26 | 29 | && git config --system --add safe.directory '*' \ |
| @@ -44,6 +47,16 @@ | ||
| 44 | 47 | npm cache clean --force; \ |
| 45 | 48 | rm -rf /root/.npm /tmp/* |
| 46 | 49 | |
| 50 | +# GitHub CLI (preinstalled on hosted runners; workflows call `gh` with GH_TOKEN). | |
| 51 | +RUN set -eux; \ | |
| 52 | + install -m 0755 -d /etc/apt/keyrings; \ | |
| 53 | + curl -fsSL https://cli.github.com/packages/githubcli-archive-keyring.gpg -o /etc/apt/keyrings/githubcli-archive-keyring.gpg; \ | |
| 54 | + echo "deb [arch=amd64 signed-by=/etc/apt/keyrings/githubcli-archive-keyring.gpg] https://cli.github.com/packages stable main" > /etc/apt/sources.list.d/github-cli.list; \ | |
| 55 | + apt-get update; \ | |
| 56 | + apt-get install -y --no-install-recommends gh; \ | |
| 57 | + apt-get clean; \ | |
| 58 | + rm -rf /var/lib/apt/lists/* /var/cache/apt/archives/* | |
| 59 | + | |
| 47 | 60 | # Docker Engine, started on first use by the /usr/local/bin/docker wrapper. Cloudflare |
| 48 | 61 | # Containers run it rootful with iptables, so bridge networks, service DNS, port |
| 49 | 62 | # publishing and builds all work like on GitHub's hosted runners. |
modifiedcontainers/runner/src/api.ts+0−32View file
| @@ -1,6 +1,3 @@ | ||
| 1 | -import { createReadStream } from "node:fs"; | |
| 2 | -import { stat } from "node:fs/promises"; | |
| 3 | -import { Readable } from "node:stream"; | |
| 4 | 1 | import type { JobCompletion, JobSpec, LogLine, StepReport } from "@diffinery/shared"; |
| 5 | 2 | |
| 6 | 3 | export class HttpError extends Error { |
| @@ -110,35 +107,6 @@ | ||
| 110 | 107 | if (!this.base) return; |
| 111 | 108 | await this.request("/complete", { method: "POST", json: c, retries: 10 }); |
| 112 | 109 | } |
| 113 | - | |
| 114 | - /** Looks up a cache entry. Returns null on miss. */ | |
| 115 | - async cacheLookup(key: string, restoreKeys: string[]): Promise<{ key: string; url: string } | null> { | |
| 116 | - const q = new URLSearchParams({ key }); | |
| 117 | - if (restoreKeys.length) q.set("restore", restoreKeys.join(",")); | |
| 118 | - const res = await this.request(`/cache?${q}`, { allow: [404, 204] }); | |
| 119 | - if (res.status === 404 || res.status === 204) return null; | |
| 120 | - const j = (await res.json()) as { key?: string; url?: string } | null; | |
| 121 | - if (!j || !j.key) return null; | |
| 122 | - return { key: j.key, url: j.url ?? `${this.base}/cache/${encodeURIComponent(j.key)}` }; | |
| 123 | - } | |
| 124 | - | |
| 125 | - /** Downloads a URL (relative URLs resolve against the API; auth sent only to the API origin). */ | |
| 126 | - async download(url: string): Promise<Response> { | |
| 127 | - const abs = new URL(url, this.base ?? undefined).toString(); | |
| 128 | - const sameOrigin = this.base ? new URL(abs).origin === new URL(this.base).origin : false; | |
| 129 | - return this.requestUrl(abs, {}, sameOrigin); | |
| 130 | - } | |
| 131 | - | |
| 132 | - async putFile(path: string, file: string, contentType = "application/gzip"): Promise<Response> { | |
| 133 | - return this.request(path, { | |
| 134 | - method: "PUT", | |
| 135 | - body: async () => { | |
| 136 | - const s = await stat(file); | |
| 137 | - return { body: Readable.toWeb(createReadStream(file)) as unknown as NonNullable<RequestInit["body"]>, length: s.size, contentType }; | |
| 138 | - }, | |
| 139 | - allow: [409], | |
| 140 | - }); | |
| 141 | - } | |
| 142 | 110 | } |
| 143 | 111 | |
| 144 | 112 | function redactUrl(u: string): string { |
modifiedcontainers/runner/src/builtins/artifacts.ts+384−59View file
| @@ -1,96 +1,421 @@ | ||
| 1 | 1 | import { randomUUID } from "node:crypto"; |
| 2 | -import { mkdir, rm, stat, writeFile } from "node:fs/promises"; | |
| 2 | +import { copyFile, mkdir, rm } from "node:fs/promises"; | |
| 3 | +import { homedir } from "node:os"; | |
| 3 | 4 | import path from "node:path"; |
| 4 | 5 | import { errMsg, failure, StepFailure } from "../job.js"; |
| 5 | -import { globPaths, inputList, truthyInput } from "../util.js"; | |
| 6 | -import { downloadTo } from "./cache.js"; | |
| 7 | -import type { Builtin } from "./index.js"; | |
| 6 | +import { RuntimeError, type RuntimeClient } from "../runtime.js"; | |
| 7 | +import { globPaths, truthyInput } from "../util.js"; | |
| 8 | +import { extractZip, sha256File, writeZip } from "../zip.js"; | |
| 9 | +import type { Builtin, BuiltinContext } from "./index.js"; | |
| 8 | 10 | |
| 9 | -function lca(dirs: string[]): string { | |
| 10 | - if (dirs.length === 0) return "/"; | |
| 11 | - let parts = dirs[0]!.split("/"); | |
| 12 | - for (const d of dirs.slice(1)) { | |
| 13 | - const p = d.split("/"); | |
| 11 | +/** | |
| 12 | + * actions/upload-artifact and actions/download-artifact (v4 and later), on the | |
| 13 | + * artifact service the official actions use: the same zip layout, names, ids, | |
| 14 | + * digests and options, so artifacts move freely between built-in and real | |
| 15 | + * actions, jobs and runs. | |
| 16 | + */ | |
| 17 | + | |
| 18 | +const MIME: Record<string, string> = { | |
| 19 | + ".txt": "text/plain", ".html": "text/html", ".htm": "text/html", ".css": "text/css", ".csv": "text/csv", ".xml": "text/xml", ".md": "text/markdown", | |
| 20 | + ".js": "application/javascript", ".mjs": "application/javascript", ".json": "application/json", | |
| 21 | + ".png": "image/png", ".jpg": "image/jpeg", ".jpeg": "image/jpeg", ".gif": "image/gif", ".svg": "image/svg+xml", ".webp": "image/webp", ".ico": "image/x-icon", ".bmp": "image/bmp", ".tiff": "image/tiff", ".tif": "image/tiff", | |
| 22 | + ".mp3": "audio/mpeg", ".wav": "audio/wav", ".ogg": "audio/ogg", ".flac": "audio/flac", ".mp4": "video/mp4", ".webm": "video/webm", ".avi": "video/x-msvideo", ".mov": "video/quicktime", | |
| 23 | + ".pdf": "application/pdf", ".zip": "application/zip", ".tar": "application/x-tar", ".gz": "application/gzip", ".rar": "application/vnd.rar", ".7z": "application/x-7z-compressed", | |
| 24 | + ".wasm": "application/wasm", ".yaml": "application/x-yaml", ".yml": "application/x-yaml", ".woff": "font/woff", ".woff2": "font/woff2", ".ttf": "font/ttf", ".otf": "font/otf", | |
| 25 | +}; | |
| 26 | + | |
| 27 | +export const mimeTypeOf = (file: string) => MIME[path.extname(file).toLowerCase()] ?? "application/octet-stream"; | |
| 28 | + | |
| 29 | +const INVALID_NAME_CHARS = ['"', ":", "<", ">", "|", "*", "?", "\r", "\n", "\\", "/"]; | |
| 30 | +const INVALID_PATH_CHARS = ['"', ":", "<", ">", "|", "*", "?", "\r", "\n"]; | |
| 31 | + | |
| 32 | +/** The action's major version (`@v4` -> 4); builtins only run for known majors. */ | |
| 33 | +function major(c: BuiltinContext): number { | |
| 34 | + return Number(/@v?(\d+)/.exec(c.uses)?.[1] ?? 99); | |
| 35 | +} | |
| 36 | + | |
| 37 | +/** The run and job ids the artifact service expects, from the runtime token's `scp` claim. */ | |
| 38 | +export function backendIds(token: string): { run: string; job: string } { | |
| 39 | + const payload = JSON.parse(Buffer.from(token.split(".")[1] ?? "", "base64url").toString("utf8")) as { scp?: string }; | |
| 40 | + for (const scope of (payload.scp ?? "").split(" ")) { | |
| 41 | + const parts = scope.split(":"); | |
| 42 | + if (parts[0] === "Actions.Results" && parts.length === 3) return { run: parts[1]!, job: parts[2]! }; | |
| 43 | + } | |
| 44 | + throw new Error("the runtime token has no Actions.Results scope"); | |
| 45 | +} | |
| 46 | + | |
| 47 | +function runtimeFor(c: BuiltinContext): RuntimeClient { | |
| 48 | + if (!c.job.runtime) throw failure("The artifact service is not available to this job."); | |
| 49 | + return c.job.runtime; | |
| 50 | +} | |
| 51 | + | |
| 52 | +// ---------------------------------------------------------------- search (like upload-artifact's findFilesToUpload) | |
| 53 | + | |
| 54 | +const hasGlob = (s: string) => /[*?[]/.test(s); | |
| 55 | + | |
| 56 | +/** @actions/glob's search path of a pattern: its longest literal directory prefix. */ | |
| 57 | +function searchPathOf(abs: string): string { | |
| 58 | + const norm = path.normalize(abs).replace(/\/+$/, "") || "/"; | |
| 59 | + if (!hasGlob(norm)) return norm; | |
| 60 | + const parts = norm.split("/"); | |
| 61 | + const lit: string[] = []; | |
| 62 | + for (const p of parts) { | |
| 63 | + if (hasGlob(p)) break; | |
| 64 | + lit.push(p); | |
| 65 | + } | |
| 66 | + return lit.join("/") || "/"; | |
| 67 | +} | |
| 68 | + | |
| 69 | +function lca(paths: string[]): string { | |
| 70 | + let parts = paths[0]!.split("/"); | |
| 71 | + for (const p of paths.slice(1)) { | |
| 72 | + const q = p.split("/"); | |
| 14 | 73 | let i = 0; |
| 15 | - while (i < parts.length && i < p.length && parts[i] === p[i]) i++; | |
| 74 | + while (i < parts.length && i < q.length && parts[i] === q[i]) i++; | |
| 16 | 75 | parts = parts.slice(0, i); |
| 17 | 76 | } |
| 18 | 77 | return parts.join("/") || "/"; |
| 19 | 78 | } |
| 20 | 79 | |
| 80 | +export function findFilesToUpload(workspace: string, searchPath: string, includeHidden: boolean): { files: string[]; root: string } { | |
| 81 | + const lines = searchPath | |
| 82 | + .split("\n") | |
| 83 | + .map((l) => l.trim()) | |
| 84 | + .filter((l) => l && !l.startsWith("#")); | |
| 85 | + const expand = (p: string) => (p === "~" || p.startsWith("~/") ? path.join(homedir(), p.slice(1)) : p); | |
| 86 | + const negations = lines.filter((l) => l.startsWith("!")).map((l) => "!" + path.resolve(workspace, expand(l.slice(1).trim()))); | |
| 87 | + const includes = lines.filter((l) => !l.startsWith("!")).map((l) => path.resolve(workspace, expand(l))); | |
| 88 | + // Search paths, minus those inside another one (as @actions/glob does). | |
| 89 | + const candidates = includes.map(searchPathOf); | |
| 90 | + const roots: string[] = []; | |
| 91 | + for (const s of candidates) { | |
| 92 | + if (roots.includes(s)) continue; | |
| 93 | + if (candidates.some((o) => o !== s && (s.startsWith(o + "/") || o === "/"))) continue; | |
| 94 | + roots.push(s); | |
| 95 | + } | |
| 96 | + const files: string[] = []; | |
| 97 | + const seen = new Set<string>(); | |
| 98 | + for (const inc of includes) { | |
| 99 | + const root = searchPathOf(inc); | |
| 100 | + for (const f of globPaths(workspace, [inc, ...negations], { filesOnly: true })) { | |
| 101 | + if (seen.has(f)) continue; | |
| 102 | + // Hidden: the search path's own name or anything below it starts with a dot. | |
| 103 | + const rel = path.relative(path.dirname(root), f).split("/"); | |
| 104 | + if (!includeHidden && rel.some((s) => s.startsWith(".") && s !== ".." && s !== ".")) continue; | |
| 105 | + seen.add(f); | |
| 106 | + files.push(f); | |
| 107 | + } | |
| 108 | + } | |
| 109 | + let root: string; | |
| 110 | + if (roots.length > 1) root = lca(roots); | |
| 111 | + else if (files.length === 1 && roots[0] === files[0]) root = path.dirname(files[0]!); | |
| 112 | + else root = roots[0] ?? workspace; | |
| 113 | + return { files, root }; | |
| 114 | +} | |
| 115 | + | |
| 116 | +// ---------------------------------------------------------------- upload | |
| 117 | + | |
| 21 | 118 | export const artifactUpload: Builtin = async (c) => { |
| 22 | - const name = c.input("name", "artifact"); | |
| 23 | - if (/[":<>|*?\\/\r\n]/.test(name)) throw failure(`The artifact name is not valid: ${name}. Contains invalid characters.`); | |
| 24 | - const patterns = inputList(c.input("path")); | |
| 25 | - if (!patterns.length) throw failure("Input required and not supplied: path"); | |
| 119 | + const searchPath = c.input("path"); | |
| 120 | + if (!searchPath) throw failure("Input required and not supplied: path"); | |
| 26 | 121 | const ifNone = c.input("if-no-files-found", "warn").toLowerCase(); |
| 27 | - const includeHidden = truthyInput(c.input("include-hidden-files"), false); | |
| 28 | - let files = globPaths(c.job.workspace, patterns, { filesOnly: true }); | |
| 29 | - if (!includeHidden) files = files.filter((f) => !path.relative(c.job.workspace, f).split("/").some((s) => s.startsWith(".") && s !== "..")); | |
| 122 | + if (!["warn", "error", "ignore"].includes(ifNone)) throw failure(`Unrecognized if-no-files-found input. Provided: ${ifNone}. Available options: warn,error,ignore`); | |
| 123 | + const archive = truthyInput(c.input("archive"), true); | |
| 124 | + const retentionRaw = c.input("retention-days"); | |
| 125 | + const retentionDays = retentionRaw ? parseInt(retentionRaw, 10) : undefined; | |
| 126 | + if (retentionDays !== undefined && Number.isNaN(retentionDays)) throw failure("Invalid retention-days"); | |
| 127 | + const levelRaw = c.input("compression-level"); | |
| 128 | + const level = levelRaw ? parseInt(levelRaw, 10) : 6; | |
| 129 | + if (Number.isNaN(level)) throw failure("Invalid compression-level"); | |
| 130 | + if (level < 0 || level > 9) throw failure("Invalid compression-level. Valid values are 0-9"); | |
| 131 | + | |
| 132 | + const { files, root } = findFilesToUpload(c.job.workspace, searchPath, truthyInput(c.input("include-hidden-files"), false)); | |
| 30 | 133 | if (files.length === 0) { |
| 31 | - const msg = `No files were found with the provided path: ${patterns.join(", ")}. No artifacts will be uploaded.`; | |
| 134 | + const msg = `No files were found with the provided path: ${searchPath}. No artifacts will be uploaded.`; | |
| 32 | 135 | if (ifNone === "error") throw failure(msg); |
| 33 | 136 | if (ifNone === "warn") c.warn(msg); |
| 34 | 137 | else c.log(msg); |
| 35 | 138 | return; |
| 36 | 139 | } |
| 37 | - if (!c.job.api.online) { | |
| 38 | - c.warn("Artifact service is not available in offline mode; skipping upload."); | |
| 39 | - return; | |
| 140 | + c.log(`With the provided path, there will be ${files.length} file${files.length === 1 ? "" : "s"} uploaded`); | |
| 141 | + if (!archive && files.length > 1) throw failure(`When 'archive' is set to false, only a single file can be uploaded. Found ${files.length} files to upload.`); | |
| 142 | + const rt = runtimeFor(c); | |
| 143 | + const name = archive ? c.input("name", "artifact") : path.basename(files[0]!); | |
| 144 | + const bad = INVALID_NAME_CHARS.find((ch) => name.includes(ch)); | |
| 145 | + if (!name || bad) throw failure(`The artifact name is not valid: ${name}. Contains the following character: ${JSON.stringify(bad ?? "")}`); | |
| 146 | + const ids = backendIds(rt.token); | |
| 147 | + | |
| 148 | + if (truthyInput(c.input("overwrite"), false)) { | |
| 149 | + try { | |
| 150 | + await rt.twirp("ArtifactService", "DeleteArtifact", { workflow_run_backend_id: ids.run, workflow_job_run_backend_id: ids.job, name }); | |
| 151 | + } catch (e) { | |
| 152 | + if (!(e instanceof RuntimeError && e.status === 404)) c.log(`Unable to delete artifact: ${errMsg(e)}`); | |
| 153 | + } | |
| 40 | 154 | } |
| 41 | - const root = lca(files.map((f) => path.dirname(f))); | |
| 42 | - const rel = files.map((f) => path.relative(root, f)); | |
| 43 | - c.log(`With the provided path, there will be ${files.length} file(s) uploaded (root: ${root})`); | |
| 44 | - const list = path.join(c.job.tempDir, `artifact-${randomUUID()}.list`); | |
| 45 | - const archive = path.join(c.job.tempDir, `artifact-${randomUUID()}.tgz`); | |
| 155 | + | |
| 156 | + const entries = files.map((f) => { | |
| 157 | + const name = path.relative(root, f); | |
| 158 | + const badChar = INVALID_PATH_CHARS.find((ch) => name.includes(ch)); | |
| 159 | + if (badChar) throw failure(`The path for one of the files in artifact is not valid: ${name}. Contains the following character: ${JSON.stringify(badChar)}`); | |
| 160 | + return { source: f, name }; | |
| 161 | + }); | |
| 162 | + | |
| 163 | + const req: Record<string, unknown> = { | |
| 164 | + workflow_run_backend_id: ids.run, | |
| 165 | + workflow_job_run_backend_id: ids.job, | |
| 166 | + name, | |
| 167 | + version: 7, | |
| 168 | + mime_type: archive ? "application/zip" : mimeTypeOf(files[0]!), | |
| 169 | + }; | |
| 170 | + if (retentionDays) { | |
| 171 | + const max = Number(process.env.GITHUB_RETENTION_DAYS ?? c.envCtx.GITHUB_RETENTION_DAYS ?? 90) || 90; | |
| 172 | + let days = retentionDays; | |
| 173 | + if (days > max) { | |
| 174 | + c.warn(`Retention days cannot be greater than the maximum allowed retention set within the repository. Using ${max} instead.`); | |
| 175 | + days = max; | |
| 176 | + } | |
| 177 | + req.expires_at = new Date(Date.now() + days * 86400_000).toISOString(); | |
| 178 | + } | |
| 179 | + | |
| 180 | + const work = path.join(c.job.tempDir, `artifact-${randomUUID()}`); | |
| 46 | 181 | try { |
| 47 | - await writeFile(list, rel.join("\0") + "\0"); | |
| 48 | - await c.run("tar", ["-czf", archive, "-C", root, "--null", "-T", list], { silent: true }); | |
| 49 | - const size = (await stat(archive)).size; | |
| 50 | - const q = new URLSearchParams(); | |
| 51 | - const retention = c.input("retention-days"); | |
| 52 | - if (retention) q.set("retention-days", retention); | |
| 53 | - if (truthyInput(c.input("overwrite"), false)) q.set("overwrite", "true"); | |
| 54 | - const qs = q.toString(); | |
| 55 | - const res = await c.job.api.putFile(`/artifacts/${encodeURIComponent(name)}${qs ? `?${qs}` : ""}`, archive); | |
| 56 | - if (res.status === 409) throw failure(`An artifact with the name '${name}' already exists. Set 'overwrite: true' to replace it.`); | |
| 57 | - let body: { id?: string | number; url?: string } = {}; | |
| 182 | + let created: { ok: boolean; signed_upload_url?: string }; | |
| 58 | 183 | try { |
| 59 | - body = (await res.json()) as typeof body; | |
| 60 | - } catch { | |
| 61 | - /* no body */ | |
| 184 | + created = await rt.twirp("ArtifactService", "CreateArtifact", req); | |
| 185 | + } catch (e) { | |
| 186 | + if (e instanceof RuntimeError && e.status === 409) throw failure(`Failed to CreateArtifact: an artifact with the name '${name}' already exists on the workflow run. Set 'overwrite: true' to replace it.`); | |
| 187 | + throw e; | |
| 62 | 188 | } |
| 63 | - c.log(`Artifact '${name}' uploaded (${size} bytes compressed).`); | |
| 64 | - if (body.id !== undefined) c.setOutput("artifact-id", String(body.id)); | |
| 65 | - if (body.url) c.setOutput("artifact-url", body.url); | |
| 189 | + if (!created.ok || !created.signed_upload_url) throw failure("CreateArtifact: response from backend was not ok"); | |
| 190 | + await mkdir(work, { recursive: true }); | |
| 191 | + let file: string; | |
| 192 | + let digest: string; | |
| 193 | + if (archive) { | |
| 194 | + file = path.join(work, `${name}.zip`); | |
| 195 | + digest = (await writeZip(file, entries, level)).sha256; | |
| 196 | + } else { | |
| 197 | + file = files[0]!; | |
| 198 | + digest = await sha256File(file); | |
| 199 | + } | |
| 200 | + c.log(`Uploading artifact: ${archive ? `${name}.zip` : name}`); | |
| 201 | + const size = await rt.upload(created.signed_upload_url, file, req.mime_type as string); | |
| 202 | + c.log("Finalizing artifact upload"); | |
| 203 | + const done = await rt.twirp<{ ok: boolean; artifact_id?: string }>("ArtifactService", "FinalizeArtifact", { | |
| 204 | + workflow_run_backend_id: ids.run, | |
| 205 | + workflow_job_run_backend_id: ids.job, | |
| 206 | + name, | |
| 207 | + size: String(size), | |
| 208 | + hash: `sha256:${digest}`, | |
| 209 | + }); | |
| 210 | + if (!done.ok || !done.artifact_id) throw failure("FinalizeArtifact: response from backend was not ok"); | |
| 211 | + c.log(`Artifact ${name} has been successfully uploaded! Final size is ${size} bytes. Artifact ID is ${done.artifact_id}`); | |
| 212 | + c.setOutput("artifact-id", done.artifact_id); | |
| 213 | + c.setOutput("artifact-digest", digest); | |
| 214 | + const env = c.toolEnv(); | |
| 215 | + const url = `${env.GITHUB_SERVER_URL}/${env.GITHUB_REPOSITORY}/actions/runs/${env.GITHUB_RUN_ID}/artifacts/${done.artifact_id}`; | |
| 216 | + c.log(`Artifact download URL: ${url}`); | |
| 217 | + c.setOutput("artifact-url", url); | |
| 66 | 218 | } catch (e) { |
| 67 | 219 | if (e instanceof StepFailure) throw e; |
| 68 | 220 | throw failure(`Failed to upload artifact '${name}': ${errMsg(e)}`); |
| 69 | 221 | } finally { |
| 70 | - await rm(list, { force: true }); | |
| 71 | - await rm(archive, { force: true }); | |
| 222 | + await rm(work, { recursive: true, force: true }); | |
| 72 | 223 | } |
| 73 | 224 | }; |
| 74 | 225 | |
| 226 | +// ---------------------------------------------------------------- download | |
| 227 | + | |
| 228 | +interface Listed { | |
| 229 | + id: number; | |
| 230 | + name: string; | |
| 231 | + size: number; | |
| 232 | + digest?: string; | |
| 233 | + createdAt?: string; | |
| 234 | +} | |
| 235 | + | |
| 236 | +/** Keeps the newest artifact of each name (the toolkit's `latest: true`). */ | |
| 237 | +function latest(list: Listed[]): Listed[] { | |
| 238 | + const out = new Map<string, Listed>(); | |
| 239 | + for (const a of [...list].sort((x, y) => y.id - x.id)) if (!out.has(a.name)) out.set(a.name, a); | |
| 240 | + return [...out.values()]; | |
| 241 | +} | |
| 242 | + | |
| 243 | +/** Minimatch-style match of an artifact name: `*`, `**`, `?`, `[...]` and `{a,b}`. */ | |
| 244 | +export function nameMatches(pattern: string, name: string): boolean { | |
| 245 | + let re = ""; | |
| 246 | + for (let i = 0; i < pattern.length; i++) { | |
| 247 | + const ch = pattern[i]!; | |
| 248 | + if (ch === "*") { | |
| 249 | + if (pattern[i + 1] === "*") { | |
| 250 | + re += ".*"; | |
| 251 | + i++; | |
| 252 | + } else re += "[^/]*"; | |
| 253 | + } else if (ch === "?") re += "[^/]"; | |
| 254 | + else if (ch === "[") { | |
| 255 | + const end = pattern.indexOf("]", i + 1); | |
| 256 | + if (end < 0) re += "\\["; | |
| 257 | + else { | |
| 258 | + re += `[${pattern.slice(i + 1, end).replace(/^!/, "^").replace(/\\/g, "\\\\")}]`; | |
| 259 | + i = end; | |
| 260 | + } | |
| 261 | + } else if (ch === "{") { | |
| 262 | + const end = pattern.indexOf("}", i + 1); | |
| 263 | + if (end < 0) re += "\\{"; | |
| 264 | + else { | |
| 265 | + re += `(?:${pattern | |
| 266 | + .slice(i + 1, end) | |
| 267 | + .split(",") | |
| 268 | + .map((s) => s.replace(/[.+^$()|\\]/g, "\\$&").replace(/\*/g, "[^/]*").replace(/\?/g, "[^/]")) | |
| 269 | + .join("|")})`; | |
| 270 | + i = end; | |
| 271 | + } | |
| 272 | + } else re += ch.replace(/[.+^$()|\\]/g, "\\$&"); | |
| 273 | + } | |
| 274 | + return new RegExp(`^${re}$`).test(name); | |
| 275 | +} | |
| 276 | + | |
| 277 | +interface Source { | |
| 278 | + list(): Promise<Listed[]>; | |
| 279 | + /** Downloads artifact `a` to `file`; returns the response's content type and file name. */ | |
| 280 | + fetch(a: Listed, file: string): Promise<{ contentType: string; fileName: string | null; url: string }>; | |
| 281 | +} | |
| 282 | + | |
| 283 | +/** Artifacts of the job's own run, through the artifact service. */ | |
| 284 | +function ownRun(c: BuiltinContext): Source { | |
| 285 | + const rt = runtimeFor(c); | |
| 286 | + const ids = backendIds(rt.token); | |
| 287 | + const base = { workflow_run_backend_id: ids.run, workflow_job_run_backend_id: ids.job }; | |
| 288 | + return { | |
| 289 | + async list() { | |
| 290 | + const r = await rt.twirp<{ artifacts?: Array<{ database_id: string; name: string; size: string; created_at?: string; digest?: string }> }>("ArtifactService", "ListArtifacts", base); | |
| 291 | + return (r.artifacts ?? []).map((a) => ({ id: Number(a.database_id), name: a.name, size: Number(a.size), createdAt: a.created_at, digest: a.digest })); | |
| 292 | + }, | |
| 293 | + async fetch(a, file) { | |
| 294 | + const { signed_url } = await rt.twirp<{ signed_url: string }>("ArtifactService", "GetSignedArtifactURL", { ...base, name: a.name }); | |
| 295 | + return { ...(await rt.download(signed_url, file)), url: signed_url }; | |
| 296 | + }, | |
| 297 | + }; | |
| 298 | +} | |
| 299 | + | |
| 300 | +/** Another run's (or repository's) artifacts, through the REST API with `github-token`. */ | |
| 301 | +function otherRun(c: BuiltinContext, token: string, repository: string, runId: string): Source { | |
| 302 | + const env = c.toolEnv(); | |
| 303 | + const api = (env.GITHUB_API_URL ?? `${c.job.spec.apiUrl.replace(/\/$/, "")}/api/v3`).replace(/\/$/, ""); | |
| 304 | + const [owner, repo] = repository.split("/"); | |
| 305 | + if (!owner || !repo) throw failure(`Invalid repository: '${repository}'. Must be in format owner/repo`); | |
| 306 | + if (!/^\d+$/.test(runId)) throw failure(`Invalid run-id: '${runId}'`); | |
| 307 | + c.job.masker.add(token); | |
| 308 | + const headers = { authorization: `Bearer ${token}`, accept: "application/vnd.github+json", "user-agent": "diffinery-runner" }; | |
| 309 | + const base = `${api}/repos/${encodeURIComponent(owner)}/${encodeURIComponent(repo)}/actions`; | |
| 310 | + return { | |
| 311 | + async list() { | |
| 312 | + const out: Listed[] = []; | |
| 313 | + for (let page = 1; page <= 50; page++) { | |
| 314 | + const res = await fetch(`${base}/runs/${runId}/artifacts?per_page=100&page=${page}`, { headers }); | |
| 315 | + if (!res.ok) throw new Error(`Listing artifacts of run ${runId} in ${repository} failed: HTTP ${res.status}`); | |
| 316 | + const j = (await res.json()) as { total_count: number; artifacts: Array<{ id: number; name: string; size_in_bytes: number; digest?: string | null; created_at?: string }> }; | |
| 317 | + for (const a of j.artifacts) out.push({ id: a.id, name: a.name, size: a.size_in_bytes, digest: a.digest ?? undefined, createdAt: a.created_at }); | |
| 318 | + if (out.length >= j.total_count || j.artifacts.length === 0) break; | |
| 319 | + } | |
| 320 | + return out; | |
| 321 | + }, | |
| 322 | + async fetch(a, file) { | |
| 323 | + const res = await fetch(`${base}/artifacts/${a.id}/zip`, { headers, redirect: "manual" }); | |
| 324 | + const location = res.headers.get("location"); | |
| 325 | + if (res.status !== 302 || !location) throw new Error(`Unable to download artifact ${a.id}: HTTP ${res.status}`); | |
| 326 | + return { ...(await runtimeFor(c).download(location, file)), url: location }; | |
| 327 | + }, | |
| 328 | + }; | |
| 329 | +} | |
| 330 | + | |
| 75 | 331 | export const artifactDownload: Builtin = async (c) => { |
| 332 | + const v = major(c); | |
| 76 | 333 | const name = c.input("name"); |
| 77 | - if (!name) { | |
| 78 | - throw failure("download-artifact without `name` (download all / `pattern`) is not supported on Diffinery runners yet; specify `name`."); | |
| 334 | + const pattern = c.input("pattern"); | |
| 335 | + const idsInput = c.input("artifact-ids"); | |
| 336 | + const merge = truthyInput(c.input("merge-multiple"), false); | |
| 337 | + const skipDecompress = truthyInput(c.input("skip-decompress"), false); | |
| 338 | + const onMismatch = (c.input("digest-mismatch") || (v >= 8 ? "error" : "warn")).toLowerCase(); | |
| 339 | + if (!["ignore", "info", "warn", "error"].includes(onMismatch)) throw failure(`Invalid value for 'digest-mismatch': '${onMismatch}'. Valid options are: ignore, info, warn, error`); | |
| 340 | + if (name && idsInput) throw failure("Inputs 'name' and 'artifact-ids' cannot be used together. Please specify only one."); | |
| 341 | + let dest = c.input("path") || c.job.workspace; | |
| 342 | + if (dest.startsWith("~")) dest = dest.replace("~", homedir()); | |
| 343 | + const resolved = path.resolve(c.job.workspace, dest); | |
| 344 | + const token = c.input("github-token"); | |
| 345 | + const source = token ? otherRun(c, token, c.input("repository", `${c.job.spec.repo.owner}/${c.job.spec.repo.name}`), c.input("run-id")) : ownRun(c); | |
| 346 | + | |
| 347 | + let artifacts: Listed[]; | |
| 348 | + try { | |
| 349 | + const all = latest(await source.list()); | |
| 350 | + if (name) { | |
| 351 | + const a = all.find((x) => x.name === name); | |
| 352 | + if (!a) throw failure(`Unable to download artifact(s): Artifact not found for name: ${name} | |
| 353 | + Please ensure that your artifact is not expired and the artifact was uploaded using a compatible version of toolkit/upload-artifact. | |
| 354 | + For more information, visit the GitHub Artifacts FAQ: https://github.com/actions/toolkit/blob/main/packages/artifact/docs/faq.md`); | |
| 355 | + artifacts = [a]; | |
| 356 | + } else if (idsInput) { | |
| 357 | + const wanted = idsInput.split(",").map((s) => s.trim()).filter(Boolean).map((s) => { | |
| 358 | + const n = parseInt(s, 10); | |
| 359 | + if (Number.isNaN(n)) throw failure(`Invalid artifact ID: '${s}'. Must be a number.`); | |
| 360 | + return n; | |
| 361 | + }); | |
| 362 | + if (!wanted.length) throw failure("No valid artifact IDs provided in 'artifact-ids' input"); | |
| 363 | + artifacts = all.filter((a) => wanted.includes(a.id)); | |
| 364 | + if (!artifacts.length) throw failure("None of the provided artifact IDs were found"); | |
| 365 | + const missing = wanted.filter((id) => !artifacts.some((a) => a.id === id)); | |
| 366 | + if (missing.length) c.warn(`Could not find the following artifact IDs: ${missing.join(", ")}`); | |
| 367 | + } else { | |
| 368 | + artifacts = pattern ? all.filter((a) => nameMatches(pattern, a.name)) : all; | |
| 369 | + if (pattern) c.log(`Filtering artifacts by pattern '${pattern}'`); | |
| 370 | + else c.log("No input name, artifact-ids or pattern filtered specified, downloading all artifacts"); | |
| 371 | + } | |
| 372 | + } catch (e) { | |
| 373 | + if (e instanceof StepFailure) throw e; | |
| 374 | + throw failure(`Unable to download artifact(s): ${errMsg(e)}`); | |
| 79 | 375 | } |
| 80 | - if (!c.job.api.online) throw failure("Artifact service is not available in offline mode."); | |
| 81 | - const dest = path.resolve(c.job.workspace, c.input("path", ".")); | |
| 82 | - await mkdir(dest, { recursive: true }); | |
| 83 | - const archive = path.join(c.job.tempDir, `artifact-${randomUUID()}.tgz`); | |
| 376 | + if (artifacts.length) { | |
| 377 | + c.log("Preparing to download the following artifacts:"); | |
| 378 | + for (const a of artifacts) c.log(`- ${a.name} (ID: ${a.id}, Size: ${a.size}, Expected Digest: ${a.digest})`); | |
| 379 | + } | |
| 380 | + | |
| 381 | + const single = !!name || merge || (v >= 5 && artifacts.length === 1); | |
| 382 | + const mismatches: string[] = []; | |
| 383 | + const work = path.join(c.job.tempDir, `download-${randomUUID()}`); | |
| 84 | 384 | try { |
| 85 | - try { | |
| 86 | - await downloadTo(c, c.job.api.url(`/artifacts/${encodeURIComponent(name)}`), archive); | |
| 87 | - } catch (e) { | |
| 88 | - throw failure(`Unable to download artifact '${name}': ${errMsg(e)}`); | |
| 385 | + await mkdir(work, { recursive: true }); | |
| 386 | + for (const a of artifacts) { | |
| 387 | + const into = single ? resolved : path.join(resolved, a.name); | |
| 388 | + await mkdir(into, { recursive: true }); | |
| 389 | + const file = path.join(work, String(a.id)); | |
| 390 | + let got: { contentType: string; fileName: string | null; url: string }; | |
| 391 | + try { | |
| 392 | + got = await source.fetch(a, file); | |
| 393 | + } catch (e) { | |
| 394 | + throw failure(`Unable to download artifact(s): ${errMsg(e)}`); | |
| 395 | + } | |
| 396 | + const digest = `sha256:${await sha256File(file)}`; | |
| 397 | + if (a.digest && a.digest !== digest) { | |
| 398 | + mismatches.push(a.name); | |
| 399 | + const msg = `Artifact '${a.name}' digest validation failed. Please verify the integrity of the artifact.`; | |
| 400 | + if (onMismatch === "warn") c.warn(msg); | |
| 401 | + else if (onMismatch === "info") c.log(msg); | |
| 402 | + } | |
| 403 | + const isZip = ["application/zip", "application/x-zip-compressed", "application/zip-compressed"].includes(got.contentType) || new URL(got.url).pathname.toLowerCase().endsWith(".zip"); | |
| 404 | + if (isZip && !skipDecompress) await extractZip(file, into); | |
| 405 | + else { | |
| 406 | + const fileName = path.basename(got.fileName || (isZip ? `${a.name}.zip` : a.name) || "artifact"); | |
| 407 | + c.log(`Downloading raw file (non-zip) to: ${path.join(into, fileName)}`); | |
| 408 | + await copyFile(file, path.join(into, fileName)); | |
| 409 | + } | |
| 410 | + await rm(file, { force: true }); | |
| 411 | + c.log(`Artifact '${a.name}' downloaded to ${into}`); | |
| 89 | 412 | } |
| 90 | - await c.run("tar", ["-xzf", archive, "-C", dest], { silent: true }); | |
| 91 | - c.log(`Artifact '${name}' downloaded to ${dest}`); | |
| 92 | - c.setOutput("download-path", dest); | |
| 93 | 413 | } finally { |
| 94 | - await rm(archive, { force: true }); | |
| 414 | + await rm(work, { recursive: true, force: true }); | |
| 95 | 415 | } |
| 416 | + if (mismatches.length && onMismatch === "error") throw failure(`Digest validation failed for artifact(s): ${mismatches.join(", ")}. Use 'digest-mismatch: warn' to continue on mismatch.`); | |
| 417 | + c.log(`Total of ${artifacts.length} artifact(s) downloaded`); | |
| 418 | + c.setOutput("download-path", resolved); | |
| 419 | + c.log("Download artifact has finished successfully"); | |
| 96 | 420 | }; |
| 421 | + |
modifiedcontainers/runner/src/builtins/cache.ts+131−68View file
| @@ -1,130 +1,193 @@ | ||
| 1 | -import { randomUUID } from "node:crypto"; | |
| 2 | -import { createWriteStream } from "node:fs"; | |
| 3 | -import { rm, stat } from "node:fs/promises"; | |
| 1 | +import { createHash, randomUUID } from "node:crypto"; | |
| 2 | +import { mkdir, rm, stat, writeFile } from "node:fs/promises"; | |
| 4 | 3 | import path from "node:path"; |
| 5 | -import { Readable } from "node:stream"; | |
| 6 | -import { pipeline } from "node:stream/promises"; | |
| 7 | -import type { ReadableStream as WebReadableStream } from "node:stream/web"; | |
| 8 | -import { errMsg, failure } from "../job.js"; | |
| 4 | +import { errMsg, failure, type JobRunner } from "../job.js"; | |
| 5 | +import { RuntimeError } from "../runtime.js"; | |
| 9 | 6 | import { globPaths, inputList, truthyInput } from "../util.js"; |
| 10 | 7 | import type { Builtin, BuiltinContext } from "./index.js"; |
| 11 | 8 | |
| 12 | -export async function downloadTo(c: BuiltinContext, url: string, file: string): Promise<void> { | |
| 13 | - const res = await c.job.api.download(url); | |
| 14 | - if (!res.body) throw new Error("empty response body"); | |
| 15 | - await pipeline(Readable.fromWeb(res.body as unknown as WebReadableStream), createWriteStream(file)); | |
| 9 | +/** | |
| 10 | + * actions/cache (and the `cache:` of setup-*), speaking the same cache service | |
| 11 | + * and archive format as the official toolkit: a `tar --posix` archive of | |
| 12 | + * workspace-relative paths, zstd-compressed when zstd is installed, keyed by | |
| 13 | + * key + version (a hash of the path list and compression). Entries saved here | |
| 14 | + * restore with the real action and vice versa. | |
| 15 | + */ | |
| 16 | + | |
| 17 | +type Compression = "zstd-without-long" | "gzip"; | |
| 18 | + | |
| 19 | +const compressionOf = new WeakMap<JobRunner, Promise<Compression>>(); | |
| 20 | + | |
| 21 | +/** Same rule as @actions/cache: zstd when `zstd` runs, else gzip. */ | |
| 22 | +export function cacheCompression(c: BuiltinContext): Promise<Compression> { | |
| 23 | + let p = compressionOf.get(c.job); | |
| 24 | + if (!p) { | |
| 25 | + p = c.exec("zstd", ["--quiet", "--version"], { capture: true, silent: true }).then( | |
| 26 | + (r) => (r.exitCode === 0 && (r.stdout + r.stderr).trim() ? "zstd-without-long" : "gzip"), | |
| 27 | + () => "gzip" as const, | |
| 28 | + ); | |
| 29 | + compressionOf.set(c.job, p); | |
| 30 | + } | |
| 31 | + return p; | |
| 16 | 32 | } |
| 17 | 33 | |
| 34 | +/** @actions/cache's version: the path patterns as given, the compression, and its salt. */ | |
| 35 | +export function cacheVersion(paths: string[], compression: Compression): string { | |
| 36 | + return createHash("sha256").update([...paths, compression, "1.0"].join("|")).digest("hex"); | |
| 37 | +} | |
| 38 | + | |
| 39 | +const archiveName = (m: Compression) => (m === "gzip" ? "cache.tgz" : "cache.tzst"); | |
| 40 | +const compressArgs = (m: Compression, mode: "create" | "extract") => (m === "gzip" ? ["-z"] : ["--use-compress-program", mode === "create" ? "zstdmt" : "unzstd"]); | |
| 41 | + | |
| 42 | +/** Case-insensitive like actions/cache's isExactKeyMatch. */ | |
| 43 | +export const isExactKeyMatch = (key: string, matched?: string) => !!matched && key.localeCompare(matched, undefined, { sensitivity: "accent" }) === 0; | |
| 44 | + | |
| 18 | 45 | export interface RestoreResult { |
| 19 | 46 | matchedKey?: string; |
| 20 | 47 | exact: boolean; |
| 21 | 48 | } |
| 22 | 49 | |
| 23 | -/** Restores a cache entry; never throws for transport errors (warns instead). */ | |
| 50 | +function runtimeOf(c: BuiltinContext) { | |
| 51 | + const rt = c.job.runtime; | |
| 52 | + if (!rt) c.warn("The cache service is not available to this job; skipping."); | |
| 53 | + return rt; | |
| 54 | +} | |
| 55 | + | |
| 56 | +/** Restores a cache entry; never throws for service errors (warns instead). */ | |
| 24 | 57 | export async function restoreCache(c: BuiltinContext, paths: string[], key: string, restoreKeys: string[], lookupOnly = false): Promise<RestoreResult> { |
| 25 | 58 | if (!key) throw failure("Input required and not supplied: key"); |
| 26 | 59 | if (paths.length === 0) throw failure("Input required and not supplied: path"); |
| 27 | - if (!c.job.api.online) { | |
| 28 | - c.warn("Cache service is not available in offline mode; skipping restore."); | |
| 29 | - return { exact: false }; | |
| 60 | + for (const k of [key, ...restoreKeys]) { | |
| 61 | + if (k.length > 512) throw failure(`Key Validation Error: ${k} cannot be larger than 512 characters.`); | |
| 62 | + if (k.includes(",")) throw failure(`Key Validation Error: ${k} cannot contain commas.`); | |
| 30 | 63 | } |
| 31 | - let hit: { key: string; url: string } | null; | |
| 64 | + const rt = runtimeOf(c); | |
| 65 | + if (!rt) return { exact: false }; | |
| 66 | + const compression = await cacheCompression(c); | |
| 67 | + let hit: { ok: boolean; signed_download_url?: string; matched_key?: string }; | |
| 32 | 68 | try { |
| 33 | - hit = await c.job.api.cacheLookup(key, restoreKeys); | |
| 69 | + hit = await rt.twirp("CacheService", "GetCacheEntryDownloadURL", { key, restore_keys: restoreKeys, version: cacheVersion(paths, compression) }); | |
| 34 | 70 | } catch (e) { |
| 35 | - c.warn(`Failed to look up cache: ${errMsg(e)}`); | |
| 71 | + c.warn(`Failed to restore: ${errMsg(e)}`); | |
| 36 | 72 | return { exact: false }; |
| 37 | 73 | } |
| 38 | - if (!hit) { | |
| 39 | - c.log(`Cache not found for input keys: ${[key, ...restoreKeys].join(", ")}`); | |
| 40 | - return { exact: false }; | |
| 41 | - } | |
| 42 | - const exact = hit.key === key; | |
| 43 | - if (lookupOnly) { | |
| 44 | - c.log(`Cache found and can be restored from key: ${hit.key}`); | |
| 45 | - return { matchedKey: hit.key, exact }; | |
| 46 | - } | |
| 47 | - const file = path.join(c.job.tempDir, `cache-${randomUUID()}.tgz`); | |
| 74 | + if (!hit.ok || !hit.matched_key || !hit.signed_download_url) return { exact: false }; | |
| 75 | + const exact = isExactKeyMatch(key, hit.matched_key); | |
| 76 | + if (lookupOnly) return { matchedKey: hit.matched_key, exact }; | |
| 77 | + const dir = path.join(c.job.tempDir, `cache-${randomUUID()}`); | |
| 78 | + const file = path.join(dir, archiveName(compression)); | |
| 48 | 79 | try { |
| 49 | - await downloadTo(c, hit.url, file); | |
| 80 | + await mkdir(dir, { recursive: true }); | |
| 81 | + await rt.download(hit.signed_download_url, file); | |
| 50 | 82 | const size = (await stat(file)).size; |
| 51 | 83 | c.log(`Cache Size: ~${Math.round(size / 1024 / 1024)} MB (${size} B)`); |
| 52 | - await c.run("tar", ["-xzPf", file], { silent: true }); | |
| 53 | - c.log(`Cache restored from key: ${hit.key}`); | |
| 54 | - return { matchedKey: hit.key, exact }; | |
| 84 | + await mkdir(c.job.workspace, { recursive: true }); | |
| 85 | + await c.run("tar", ["-xf", file, "-P", "-C", c.job.workspace, ...compressArgs(compression, "extract")], { silent: true }); | |
| 86 | + return { matchedKey: hit.matched_key, exact }; | |
| 55 | 87 | } catch (e) { |
| 56 | - c.warn(`Failed to restore cache: ${errMsg(e)}`); | |
| 88 | + c.warn(`Failed to restore: ${errMsg(e)}`); | |
| 57 | 89 | return { exact: false }; |
| 58 | 90 | } finally { |
| 59 | - await rm(file, { force: true }); | |
| 91 | + await rm(dir, { recursive: true, force: true }); | |
| 60 | 92 | } |
| 61 | 93 | } |
| 62 | 94 | |
| 63 | -/** Archives `paths` (tar.gz, absolute paths) and uploads it under `key`. */ | |
| 95 | +/** Archives `paths` and saves it under `key`. Never throws for service errors. */ | |
| 64 | 96 | export async function saveCache(c: BuiltinContext, paths: string[], key: string): Promise<boolean> { |
| 65 | - if (!c.job.api.online) { | |
| 66 | - c.warn("Cache service is not available in offline mode; skipping save."); | |
| 67 | - return false; | |
| 68 | - } | |
| 69 | - const resolved = globPaths(c.job.workspace, paths); | |
| 97 | + const rt = runtimeOf(c); | |
| 98 | + if (!rt) return false; | |
| 99 | + const ws = c.job.workspace; | |
| 100 | + const resolved = globPaths(ws, paths).map((f) => path.relative(ws, f) || "."); | |
| 70 | 101 | if (resolved.length === 0) { |
| 71 | 102 | c.warn("Path Validation Error: Path(s) specified in the action for caching do(es) not exist, hence no cache is being saved."); |
| 72 | 103 | return false; |
| 73 | 104 | } |
| 74 | - const file = path.join(c.job.tempDir, `cache-${randomUUID()}.tgz`); | |
| 105 | + const compression = await cacheCompression(c); | |
| 106 | + const dir = path.join(c.job.tempDir, `cache-${randomUUID()}`); | |
| 107 | + const name = archiveName(compression); | |
| 75 | 108 | try { |
| 76 | - await c.run("tar", ["-czPf", file, "--", ...resolved], { silent: true }); | |
| 109 | + await mkdir(dir, { recursive: true }); | |
| 110 | + await writeFile(path.join(dir, "manifest.txt"), resolved.join("\n")); | |
| 111 | + await c.run("tar", ["--posix", "-cf", name, "--exclude", name, "-P", "-C", ws, "--files-from", "manifest.txt", ...compressArgs(compression, "create")], { cwd: dir, silent: true }); | |
| 112 | + const file = path.join(dir, name); | |
| 77 | 113 | const size = (await stat(file)).size; |
| 78 | 114 | c.log(`Cache Size: ~${Math.round(size / 1024 / 1024)} MB (${size} B)`); |
| 79 | - const res = await c.job.api.putFile(`/cache/${encodeURIComponent(key)}`, file); | |
| 80 | - if (res.status === 409) { | |
| 81 | - c.log(`Cache entry with key ${key} already exists; not saving.`); | |
| 115 | + if (size > 10 * 1024 ** 3) { | |
| 116 | + c.warn(`Cache size of ~${Math.round(size / 1024 / 1024)} MB (${size} B) is over the 10GB limit, not saving cache.`); | |
| 82 | 117 | return false; |
| 83 | 118 | } |
| 119 | + const version = cacheVersion(paths, compression); | |
| 120 | + let created: { ok: boolean; signed_upload_url?: string; message?: string }; | |
| 121 | + try { | |
| 122 | + created = await rt.twirp("CacheService", "CreateCacheEntry", { key, version }); | |
| 123 | + } catch (e) { | |
| 124 | + if (e instanceof RuntimeError && e.status === 409) { | |
| 125 | + c.warn(`Failed to save: Unable to reserve cache with key ${key}, another job may be creating this cache.`); | |
| 126 | + return false; | |
| 127 | + } | |
| 128 | + throw e; | |
| 129 | + } | |
| 130 | + if (!created.ok || !created.signed_upload_url) throw new Error(created.message || `Unable to reserve cache with key ${key}`); | |
| 131 | + await rt.upload(created.signed_upload_url, file); | |
| 132 | + const done = await rt.twirp<{ ok: boolean; message?: string }>("CacheService", "FinalizeCacheEntryUpload", { key, version, size_bytes: String(size) }); | |
| 133 | + if (!done.ok) throw new Error(done.message || `Unable to finalize cache with key ${key}`); | |
| 84 | 134 | c.log(`Cache saved with key: ${key}`); |
| 85 | 135 | return true; |
| 86 | 136 | } catch (e) { |
| 87 | - c.warn(`Failed to save cache: ${errMsg(e)}`); | |
| 137 | + c.warn(`Failed to save: ${errMsg(e)}`); | |
| 88 | 138 | return false; |
| 89 | 139 | } finally { |
| 90 | - await rm(file, { force: true }); | |
| 140 | + await rm(dir, { recursive: true, force: true }); | |
| 91 | 141 | } |
| 92 | 142 | } |
| 93 | 143 | |
| 94 | -export const cacheAction: Builtin = async (c) => { | |
| 144 | +/** actions/cache's path list: one per line, `! pattern` normalised to `!pattern`. */ | |
| 145 | +const pathInput = (c: BuiltinContext) => inputList(c.input("path")).map((p) => p.replace(/^!\s+/, "!")); | |
| 146 | + | |
| 147 | +function restoreInputs(c: BuiltinContext) { | |
| 95 | 148 | const key = c.input("key"); |
| 96 | - const paths = inputList(c.input("path")); | |
| 97 | - const restoreKeys = inputList(c.input("restore-keys")); | |
| 98 | - const r = await restoreCache(c, paths, key, restoreKeys, truthyInput(c.input("lookup-only"), false)); | |
| 99 | - c.setOutput("cache-hit", r.exact ? "true" : "false"); | |
| 100 | - c.setOutput("cache-primary-key", key); | |
| 101 | - c.setOutput("cache-matched-key", r.matchedKey ?? ""); | |
| 102 | - if (!r.matchedKey && truthyInput(c.input("fail-on-cache-miss"), false)) { | |
| 103 | - throw failure(`Failed to restore cache entry. Exiting as fail-on-cache-miss is set. Input key: ${key}`); | |
| 149 | + if (!key) throw failure("Input required and not supplied: key"); | |
| 150 | + const paths = pathInput(c); | |
| 151 | + if (!paths.length) throw failure("Input required and not supplied: path"); | |
| 152 | + return { key, paths, restoreKeys: inputList(c.input("restore-keys")), lookupOnly: truthyInput(c.input("lookup-only"), false), failOnMiss: truthyInput(c.input("fail-on-cache-miss"), false) }; | |
| 153 | +} | |
| 154 | + | |
| 155 | +async function restoreStep(c: BuiltinContext, o: ReturnType<typeof restoreInputs>): Promise<RestoreResult> { | |
| 156 | + const r = await restoreCache(c, o.paths, o.key, o.restoreKeys, o.lookupOnly); | |
| 157 | + if (!r.matchedKey) { | |
| 158 | + if (o.failOnMiss) throw failure(`Failed to restore cache entry. Exiting as fail-on-cache-miss is set. Input key: ${o.key}`); | |
| 159 | + c.log(`Cache not found for input keys: ${[o.key, ...o.restoreKeys].join(", ")}`); | |
| 160 | + return r; | |
| 104 | 161 | } |
| 105 | - if (r.exact) return; | |
| 162 | + c.setOutput("cache-hit", r.exact ? "true" : "false"); | |
| 163 | + c.log(o.lookupOnly ? `Cache found and can be restored from key: ${r.matchedKey}` : `Cache restored from key: ${r.matchedKey}`); | |
| 164 | + return r; | |
| 165 | +} | |
| 166 | + | |
| 167 | +export const cacheAction: Builtin = async (c) => { | |
| 168 | + const o = restoreInputs(c); | |
| 169 | + const r = await restoreStep(c, o); | |
| 106 | 170 | const saveAlways = truthyInput(c.input("save-always"), false); |
| 107 | 171 | c.registerPost(`Post ${c.name}`, saveAlways ? "always()" : "success()", async (pc) => { |
| 108 | - await saveCache(pc, paths, key); | |
| 172 | + if (isExactKeyMatch(o.key, r.matchedKey)) { | |
| 173 | + pc.log(`Cache hit occurred on the primary key ${o.key}, not saving cache.`); | |
| 174 | + return; | |
| 175 | + } | |
| 176 | + await saveCache(pc, o.paths, o.key); | |
| 109 | 177 | }); |
| 110 | 178 | }; |
| 111 | 179 | |
| 112 | 180 | export const cacheRestoreAction: Builtin = async (c) => { |
| 113 | - const key = c.input("key"); | |
| 114 | - const paths = inputList(c.input("path")); | |
| 115 | - const r = await restoreCache(c, paths, key, inputList(c.input("restore-keys")), truthyInput(c.input("lookup-only"), false)); | |
| 116 | - c.setOutput("cache-hit", r.exact ? "true" : "false"); | |
| 117 | - c.setOutput("cache-primary-key", key); | |
| 118 | - c.setOutput("cache-matched-key", r.matchedKey ?? ""); | |
| 119 | - if (!r.matchedKey && truthyInput(c.input("fail-on-cache-miss"), false)) { | |
| 120 | - throw failure(`Failed to restore cache entry. Exiting as fail-on-cache-miss is set. Input key: ${key}`); | |
| 121 | - } | |
| 181 | + const o = restoreInputs(c); | |
| 182 | + c.setOutput("cache-primary-key", o.key); | |
| 183 | + const r = await restoreStep(c, o); | |
| 184 | + if (r.matchedKey) c.setOutput("cache-matched-key", r.matchedKey); | |
| 122 | 185 | }; |
| 123 | 186 | |
| 124 | 187 | export const cacheSaveAction: Builtin = async (c) => { |
| 125 | 188 | const key = c.input("key"); |
| 126 | 189 | if (!key) throw failure("Input required and not supplied: key"); |
| 127 | - const paths = inputList(c.input("path")); | |
| 190 | + const paths = pathInput(c); | |
| 128 | 191 | if (!paths.length) throw failure("Input required and not supplied: path"); |
| 129 | 192 | await saveCache(c, paths, key); |
| 130 | 193 | }; |
modifiedcontainers/runner/src/builtins/checkout.ts+70−27View file
| @@ -32,25 +32,50 @@ | ||
| 32 | 32 | const spec = c.job.spec; |
| 33 | 33 | const ownRepo = `${spec.repo.owner}/${spec.repo.name}`; |
| 34 | 34 | const repository = c.input("repository", ownRepo); |
| 35 | + if (!/^[\w.-]+\/[\w.-]+$/.test(repository)) throw failure(`Invalid repository '${repository}'. Expected format {owner}/{repo}.`); | |
| 35 | 36 | const isOwn = repository.toLowerCase() === ownRepo.toLowerCase(); |
| 36 | - const token = c.input("token", spec.token); | |
| 37 | 37 | const refInput = c.input("ref"); |
| 38 | 38 | const depthRaw = Number(c.input("fetch-depth", "1")); |
| 39 | 39 | const fetchDepth = Number.isFinite(depthRaw) && depthRaw >= 0 ? Math.floor(depthRaw) : 1; |
| 40 | 40 | const dest = path.resolve(c.job.workspace, c.input("path", ".")); |
| 41 | + if (dest !== c.job.workspace && !dest.startsWith(c.job.workspace + path.sep)) throw failure(`Repository path '${dest}' is not under '${c.job.workspace}'`); | |
| 41 | 42 | const clean = truthyInput(c.input("clean"), true); |
| 42 | 43 | const submodules = c.input("submodules", "false").toLowerCase(); |
| 43 | 44 | const persist = truthyInput(c.input("persist-credentials"), true); |
| 44 | 45 | const fetchTags = truthyInput(c.input("fetch-tags"), false); |
| 45 | - if (truthyInput(c.input("lfs"), false)) c.warn("Git LFS is not supported by the Diffinery checkout built-in; LFS pointers will not be resolved."); | |
| 46 | - if (c.input("sparse-checkout")) c.warn("sparse-checkout is not supported by the Diffinery checkout built-in; doing a full checkout."); | |
| 46 | + const showProgress = truthyInput(c.input("show-progress"), true); | |
| 47 | + const lfs = truthyInput(c.input("lfs"), false); | |
| 48 | + const sparse = c.input("sparse-checkout").split("\n").map((l) => l.trim()).filter(Boolean); | |
| 49 | + const coneMode = truthyInput(c.input("sparse-checkout-cone-mode"), true); | |
| 50 | + // Like actions/checkout, sparse checkouts fetch without blobs unless a filter is given. | |
| 51 | + const filter = c.input("filter") || (sparse.length ? "blob:none" : ""); | |
| 47 | 52 | |
| 48 | - const url = cloneUrlFor(spec, repository); | |
| 49 | - const header = basicAuthHeader(token); | |
| 50 | - c.job.masker.add(token); | |
| 51 | - c.job.masker.add(header.slice("AUTHORIZATION: basic ".length)); | |
| 52 | - const authArgs = isHttp(url) && token ? ["-c", `http.extraheader=${header}`] : []; | |
| 53 | - const env = c.toolEnv({ GIT_TERMINAL_PROMPT: "0", GIT_LFS_SKIP_SMUDGE: "1" }); | |
| 53 | + const server = c.input("github-server-url").replace(/\/+$/, ""); | |
| 54 | + const ownOrigin = isHttp(spec.repo.cloneUrl) ? new URL(spec.repo.cloneUrl).origin : ""; | |
| 55 | + const url = server && server !== ownOrigin ? `${server}/${repository}.git` : cloneUrlFor(spec, repository); | |
| 56 | + const origin = isHttp(url) ? new URL(url).origin : ""; | |
| 57 | + // The job's own token only ever goes to this server; any other host needs an explicit `token:`. | |
| 58 | + const token = c.input("token") || (origin && origin === ownOrigin ? spec.token : ""); | |
| 59 | + const header = token ? basicAuthHeader(token) : ""; | |
| 60 | + if (token) { | |
| 61 | + c.job.masker.add(token); | |
| 62 | + c.job.masker.add(header.slice("AUTHORIZATION: basic ".length)); | |
| 63 | + } | |
| 64 | + const authKey = `http.${origin}/.extraheader`; | |
| 65 | + const env = c.toolEnv({ GIT_TERMINAL_PROMPT: "0", ...(lfs ? {} : { GIT_LFS_SKIP_SMUDGE: "1" }) }); | |
| 66 | + /** Configuration for the git commands run here (and the git, git-lfs and submodule processes they start). */ | |
| 67 | + const withConfig = (key: string, value: string) => { | |
| 68 | + const n = Number(env.GIT_CONFIG_COUNT) || 0; | |
| 69 | + Object.assign(env, { GIT_CONFIG_COUNT: String(n + 1), [`GIT_CONFIG_KEY_${n}`]: key, [`GIT_CONFIG_VALUE_${n}`]: value }); | |
| 70 | + }; | |
| 71 | + // Like actions/checkout's temporary global config: trust the checkout for these git commands only. | |
| 72 | + if (truthyInput(c.input("set-safe-directory"), true)) withConfig("safe.directory", dest); | |
| 73 | + // The token reaches git through the environment while fetching, so a checkout that fails half | |
| 74 | + // way leaves nothing in .git/config; it is written there at the end, only to persist it. | |
| 75 | + if (origin && header) { | |
| 76 | + withConfig(authKey, header); | |
| 77 | + withConfig(`url.${origin}/.insteadOf`, `git@${new URL(origin).host}:`); | |
| 78 | + } | |
| 54 | 79 | const git = (args: string[], o: { silent?: boolean; allowFail?: boolean } = {}) => |
| 55 | 80 | o.allowFail ? c.exec("git", args, { cwd: dest, env, capture: true, silent: o.silent }) : c.run("git", args, { cwd: dest, env, silent: o.silent }); |
| 56 | 81 | |
| @@ -71,6 +96,9 @@ | ||
| 71 | 96 | const setUrl = await git(["remote", "set-url", "origin", url], { allowFail: true, silent: true }); |
| 72 | 97 | if (setUrl.exitCode !== 0) await git(["remote", "add", "origin", url]); |
| 73 | 98 | await git(["config", "--local", "gc.auto", "0"]); |
| 99 | + // A reused workspace may still hold another job's credentials; they would be sent alongside ours. | |
| 100 | + if (origin) await git(["config", "--local", "--unset-all", authKey], { allowFail: true, silent: true }); | |
| 101 | + if (lfs) await git(["lfs", "install", "--local"]); | |
| 74 | 102 | |
| 75 | 103 | // Decide what to fetch |
| 76 | 104 | let sha: string | undefined; |
| @@ -87,8 +115,8 @@ | ||
| 87 | 115 | |
| 88 | 116 | const depthArgs = fetchDepth > 0 ? [`--depth=${fetchDepth}`] : []; |
| 89 | 117 | const tagArgs = fetchDepth === 0 || fetchTags ? [] : ["--no-tags"]; |
| 90 | - const fetch = (refspecs: string[], allowFail = false) => | |
| 91 | - git([...authArgs, "fetch", ...tagArgs, "--prune", "--no-recurse-submodules", ...depthArgs, "origin", ...refspecs], { allowFail }); | |
| 118 | + const extraArgs = [...(filter ? [`--filter=${filter}`] : []), ...(showProgress ? ["--progress"] : [])]; | |
| 119 | + const fetch = (refspecs: string[], allowFail = false) => git(["fetch", ...tagArgs, "--prune", "--no-recurse-submodules", ...extraArgs, ...depthArgs, "origin", ...refspecs], { allowFail }); | |
| 92 | 120 | |
| 93 | 121 | let checkoutTarget: string | undefined; |
| 94 | 122 | let branch: string | undefined; |
| @@ -131,8 +159,20 @@ | ||
| 131 | 159 | checkoutTarget ??= sha ?? "FETCH_HEAD"; |
| 132 | 160 | } |
| 133 | 161 | |
| 134 | - if (branch) await git(["checkout", "--progress", "--force", "-B", branch, checkoutTarget!]); | |
| 135 | - else await git(["checkout", "--progress", "--force", checkoutTarget!]); | |
| 162 | + if (lfs) { | |
| 163 | + c.log("Fetching LFS objects"); | |
| 164 | + await git(["lfs", "fetch", "origin", checkoutTarget!]); | |
| 165 | + } | |
| 166 | + if (sparse.length) { | |
| 167 | + c.log(`Setting up sparse checkout${coneMode ? "" : " (non-cone mode)"}`); | |
| 168 | + await git(["sparse-checkout", "set", ...(coneMode ? [] : ["--no-cone"]), "--", ...sparse]); | |
| 169 | + } else if (hasGit) { | |
| 170 | + await git(["sparse-checkout", "disable"], { allowFail: true, silent: true }); | |
| 171 | + } | |
| 172 | + | |
| 173 | + const progress = showProgress ? ["--progress"] : []; | |
| 174 | + if (branch) await git(["checkout", ...progress, "--force", "-B", branch, checkoutTarget!]); | |
| 175 | + else await git(["checkout", ...progress, "--force", checkoutTarget!]); | |
| 136 | 176 | const head = (await git(["rev-parse", "HEAD"], { silent: true })).stdout.trim(); |
| 137 | 177 | if (sha && /^[0-9a-f]{40}$/i.test(sha) && head.toLowerCase() !== sha.toLowerCase()) { |
| 138 | 178 | throw failure(`Checked out ${head} but expected ${sha}.`); |
| @@ -142,24 +182,27 @@ | ||
| 142 | 182 | c.setOutput("ref", ref && ref !== "HEAD" ? ref : branch ? `refs/heads/${branch}` : ""); |
| 143 | 183 | |
| 144 | 184 | if (submodules === "true" || submodules === "recursive") { |
| 185 | + const recursive = submodules === "recursive" ? ["--recursive"] : []; | |
| 145 | 186 | c.log("Fetching submodules"); |
| 146 | - await git(["submodule", "sync", ...(submodules === "recursive" ? ["--recursive"] : [])], { allowFail: true }); | |
| 147 | - const r = await git( | |
| 148 | - [...authArgs, "submodule", "update", "--init", "--force", ...(fetchDepth > 0 ? [`--depth=${fetchDepth}`] : []), ...(submodules === "recursive" ? ["--recursive"] : [])], | |
| 149 | - { allowFail: true }, | |
| 150 | - ); | |
| 151 | - if (r.exitCode !== 0) c.warn("Failed to update submodules (best effort)."); | |
| 187 | + await git(["submodule", "sync", ...recursive]); | |
| 188 | + await git(["submodule", "update", "--init", "--force", ...(fetchDepth > 0 ? [`--depth=${fetchDepth}`] : []), ...recursive]); | |
| 189 | + if (lfs) await git(["submodule", "foreach", ...recursive, "git lfs pull"]); | |
| 152 | 190 | } |
| 153 | 191 | |
| 154 | - if (persist && authArgs.length) { | |
| 155 | - const key = `http.${new URL(url).origin}/.extraheader`; | |
| 156 | - await git(["config", "--local", key, header], { silent: true }); | |
| 157 | - c.registerPost(`Post ${c.name}`, "always()", async (pc: BuiltinContext) => { | |
| 158 | - if (!existsSync(path.join(dest, ".git"))) return; | |
| 159 | - pc.log("Removing persisted credentials"); | |
| 160 | - await pc.exec("git", ["config", "--local", "--unset-all", key], { cwd: dest, capture: true, silent: true }); | |
| 161 | - }); | |
| 192 | + if (!origin || !header || !persist) return; | |
| 193 | + // persist-credentials: later steps' git commands authenticate; the post step removes it again. | |
| 194 | + await git(["config", "--local", authKey, header], { silent: true }); | |
| 195 | + if (submodules === "true" || submodules === "recursive") { | |
| 196 | + await git(["submodule", "foreach", ...(submodules === "recursive" ? ["--recursive"] : []), `git config --local '${authKey}' '${header}'`], { silent: true }); | |
| 162 | 197 | } |
| 198 | + c.registerPost(`Post ${c.name}`, "always()", async (pc: BuiltinContext) => { | |
| 199 | + if (!existsSync(path.join(dest, ".git"))) return; | |
| 200 | + pc.log("Removing persisted credentials"); | |
| 201 | + await pc.exec("git", ["config", "--local", "--unset-all", authKey], { cwd: dest, capture: true, silent: true }); | |
| 202 | + if (submodules === "true" || submodules === "recursive") { | |
| 203 | + await pc.exec("git", ["submodule", "foreach", "--recursive", `git config --local --unset-all '${authKey}' || :`], { cwd: dest, capture: true, silent: true }); | |
| 204 | + } | |
| 205 | + }); | |
| 163 | 206 | }; |
| 164 | 207 | |
| 165 | 208 | function refspecsFor(ref: string): Array<{ refspec: string; local: string; branch?: string }> { |
modifiedcontainers/runner/src/builtins/index.ts+63−17View file
| @@ -34,26 +34,72 @@ | ||
| 34 | 34 | |
| 35 | 35 | export type Builtin = (c: BuiltinContext) => Promise<void>; |
| 36 | 36 | |
| 37 | -const githubScript: Builtin = async () => { | |
| 38 | - throw failure("actions/github-script is not supported on Diffinery runners yet. Use a `run:` step with `node` or `curl` against $GITHUB_API_URL instead."); | |
| 39 | -}; | |
| 37 | +interface BuiltinDef { | |
| 38 | + run: Builtin; | |
| 39 | + /** Major versions (`@vN`) the built-in mirrors; other refs (SHAs, branches, newer majors) run the real action. */ | |
| 40 | + majors: [number, number]; | |
| 41 | + /** The `with:` inputs it understands; anything else runs the real action. */ | |
| 42 | + inputs: string[]; | |
| 43 | + /** Further input values it cannot handle (the real action runs instead). */ | |
| 44 | + supports?: (inputs: Map<string, string>) => boolean; | |
| 45 | +} | |
| 40 | 46 | |
| 41 | -const BUILTINS: Record<string, Builtin> = { | |
| 42 | - "actions/checkout": checkout, | |
| 43 | - "actions/cache": cacheAction, | |
| 44 | - "actions/cache/restore": cacheRestoreAction, | |
| 45 | - "actions/cache/save": cacheSaveAction, | |
| 46 | - "actions/upload-artifact": artifactUpload, | |
| 47 | - "actions/download-artifact": artifactDownload, | |
| 48 | - "actions/setup-node": setupNode, | |
| 49 | - "actions/setup-python": setupPython, | |
| 50 | - "actions/setup-go": setupGo, | |
| 51 | - "actions/github-script": githubScript, | |
| 47 | +const hostArch = (v: string | undefined) => !v || v.toLowerCase() === (process.arch === "arm64" ? "arm64" : "x64"); | |
| 48 | +const CACHE_INPUTS = ["key", "path", "restore-keys", "upload-chunk-size", "enablecrossosarchive", "fail-on-cache-miss", "lookup-only", "save-always"]; | |
| 49 | + | |
| 50 | +/** | |
| 51 | + * Actions the runner implements itself because that is much faster than | |
| 52 | + * downloading and starting them. Each mirrors the official action's inputs, | |
| 53 | + * outputs and storage (the same cache and artifact services), and steps | |
| 54 | + * aside for the real action when it sees a ref or input it does not mirror. | |
| 55 | + */ | |
| 56 | +const BUILTINS: Record<string, BuiltinDef> = { | |
| 57 | + "actions/checkout": { | |
| 58 | + run: checkout, | |
| 59 | + majors: [2, 6], | |
| 60 | + inputs: ["repository", "ref", "token", "ssh-key", "ssh-known-hosts", "ssh-strict", "ssh-user", "persist-credentials", "path", "clean", "filter", "sparse-checkout", "sparse-checkout-cone-mode", "fetch-depth", "fetch-tags", "show-progress", "lfs", "submodules", "set-safe-directory", "github-server-url"], | |
| 61 | + supports: (i) => !i.get("ssh-key"), | |
| 62 | + }, | |
| 63 | + "actions/cache": { run: cacheAction, majors: [3, 6], inputs: CACHE_INPUTS }, | |
| 64 | + "actions/cache/restore": { run: cacheRestoreAction, majors: [3, 6], inputs: CACHE_INPUTS }, | |
| 65 | + "actions/cache/save": { run: cacheSaveAction, majors: [3, 6], inputs: CACHE_INPUTS }, | |
| 66 | + "actions/upload-artifact": { run: artifactUpload, majors: [4, 7], inputs: ["name", "path", "if-no-files-found", "retention-days", "compression-level", "overwrite", "include-hidden-files", "archive"] }, | |
| 67 | + "actions/download-artifact": { run: artifactDownload, majors: [4, 8], inputs: ["name", "path", "pattern", "merge-multiple", "github-token", "repository", "run-id", "artifact-ids", "skip-decompress", "digest-mismatch"] }, | |
| 68 | + "actions/setup-node": { | |
| 69 | + run: setupNode, | |
| 70 | + majors: [3, 7], | |
| 71 | + inputs: ["node-version", "node-version-file", "architecture", "check-latest", "registry-url", "scope", "token", "cache", "cache-dependency-path", "always-auth", "package-manager-cache"], | |
| 72 | + supports: (i) => hostArch(i.get("architecture")) && ["", "npm", "pnpm", "yarn"].includes((i.get("cache") ?? "").toLowerCase()), | |
| 73 | + }, | |
| 74 | + "actions/setup-python": { | |
| 75 | + run: setupPython, | |
| 76 | + majors: [4, 6], | |
| 77 | + inputs: ["python-version", "python-version-file", "cache", "architecture", "check-latest", "token", "cache-dependency-path", "update-environment", "allow-prereleases"], | |
| 78 | + supports: (i) => | |
| 79 | + hostArch(i.get("architecture")) && | |
| 80 | + !/^(pypy|graalpy)/i.test(i.get("python-version") ?? "") && | |
| 81 | + ["", "pip", "pipenv"].includes((i.get("cache") ?? "").toLowerCase()) && | |
| 82 | + (i.get("update-environment") ?? "true").toLowerCase() !== "false", | |
| 83 | + }, | |
| 84 | + "actions/setup-go": { run: setupGo, majors: [4, 6], inputs: ["go-version", "go-version-file", "check-latest", "token", "cache", "cache-dependency-path", "architecture"], supports: (i) => hostArch(i.get("architecture")) }, | |
| 52 | 85 | }; |
| 53 | 86 | |
| 54 | -export function getBuiltin(name: string, _ref: string): Builtin | null { | |
| 55 | - if (process.env.DIFFINERY_DISABLE_BUILTINS === "1") return null; | |
| 56 | - return BUILTINS[name.toLowerCase()] ?? null; | |
| 87 | +const OFF = /^(0|false|no|off)$/i; | |
| 88 | + | |
| 89 | +/** | |
| 90 | + * The built-in for `name@ref` with these `with:` inputs, or null to run the | |
| 91 | + * real action. `DIFFINERY_BUILTINS: false` in a workflow, job or step `env:` | |
| 92 | + * (or DIFFINERY_DISABLE_BUILTINS=1 on the runner) always runs the real action. | |
| 93 | + */ | |
| 94 | +export function getBuiltin(name: string, ref: string, inputs: Record<string, string> = {}, env: Record<string, string> = {}): Builtin | null { | |
| 95 | + if (process.env.DIFFINERY_DISABLE_BUILTINS === "1" || OFF.test(env.DIFFINERY_BUILTINS ?? "")) return null; | |
| 96 | + const def = BUILTINS[name.toLowerCase()]; | |
| 97 | + const m = /^v(\d+)(\.\d+){0,2}$/.exec(ref); | |
| 98 | + if (!def || !m || Number(m[1]) < def.majors[0] || Number(m[1]) > def.majors[1]) return null; | |
| 99 | + const given = new Map(Object.entries(inputs).map(([k, v]) => [k.toLowerCase(), v])); | |
| 100 | + if ([...given.keys()].some((k) => !def.inputs.includes(k))) return null; | |
| 101 | + if (def.supports && !def.supports(given)) return null; | |
| 102 | + return def.run; | |
| 57 | 103 | } |
| 58 | 104 | |
| 59 | 105 | export function makeBuiltinContext(job: JobRunner, x: StepExec, inputs: Record<string, string>, uses: string): BuiltinContext { |
modifiedcontainers/runner/src/builtins/setup-go.ts+10−8View file
| @@ -1,9 +1,8 @@ | ||
| 1 | -import { existsSync } from "node:fs"; | |
| 2 | 1 | import path from "node:path"; |
| 3 | 2 | import { errMsg, failure } from "../job.js"; |
| 4 | 3 | import { maxSatisfying, satisfies, truthyInput } from "../util.js"; |
| 5 | 4 | import type { Builtin } from "./index.js"; |
| 6 | -import { fetchJson, installTarball, isCached, packageCache, probe, readVersionFile, toolArch, toolDir } from "./tools.js"; | |
| 5 | +import { fetchJson, installTarball, isCached, packageCache, probe, readVersionFile, rootLockHash, toolArch, toolDir } from "./tools.js"; | |
| 7 | 6 | |
| 8 | 7 | interface GoRelease { |
| 9 | 8 | version: string; // "go1.22.3" |
| @@ -55,18 +54,21 @@ | ||
| 55 | 54 | if (version) c.setOutput("go-version", version); |
| 56 | 55 | |
| 57 | 56 | if (truthyInput(c.input("cache"), true)) { |
| 58 | - const deps = c.input("cache-dependency-path"); | |
| 59 | - if (!deps && !existsSync(path.join(c.job.workspace, "go.sum"))) { | |
| 60 | - c.log("go.sum not found; skipping dependency cache."); | |
| 57 | + let hash: string; | |
| 58 | + try { | |
| 59 | + hash = rootLockHash(c, c.input("cache-dependency-path"), ["go.sum"]); | |
| 60 | + if (!hash) throw failure("Some specified paths were not resolved, unable to cache dependencies."); | |
| 61 | + } catch (e) { | |
| 62 | + c.warn(`Restore cache failed: ${errMsg(e)}`); | |
| 61 | 63 | return; |
| 62 | 64 | } |
| 63 | 65 | const env = c.toolEnv(); |
| 64 | 66 | const modcache = await probe(c, "go", ["env", "GOMODCACHE"], env); |
| 65 | 67 | const buildcache = await probe(c, "go", ["env", "GOCACHE"], env); |
| 68 | + const imageOs = env.ImageOS ?? "ubuntu24"; | |
| 66 | 69 | await packageCache(c, { |
| 67 | - prefix: `setup-go-Linux-${toolArch("go")}-go-${version}`, | |
| 68 | - paths: [modcache ?? "", buildcache ?? ""], | |
| 69 | - lockPatterns: ["**/go.sum"], | |
| 70 | + key: `setup-go-Linux-${process.arch}-${imageOs}-go-${version}-${hash}`, | |
| 71 | + paths: [modcache ?? "", buildcache ?? ""].filter(Boolean), | |
| 70 | 72 | what: "go", |
| 71 | 73 | }); |
| 72 | 74 | } |
modifiedcontainers/runner/src/builtins/setup-node.ts+59−29View file
| @@ -1,8 +1,10 @@ | ||
| 1 | +import { existsSync, readFileSync } from "node:fs"; | |
| 2 | +import { mkdir, readFile, writeFile } from "node:fs/promises"; | |
| 1 | 3 | import path from "node:path"; |
| 2 | 4 | import { errMsg, failure } from "../job.js"; |
| 3 | 5 | import { maxSatisfying, parseVer, satisfies, truthyInput } from "../util.js"; |
| 4 | -import type { Builtin } from "./index.js"; | |
| 5 | -import { fetchJson, installTarball, isCached, packageCache, probe, readVersionFile, toolArch, toolDir } from "./tools.js"; | |
| 6 | +import type { Builtin, BuiltinContext } from "./index.js"; | |
| 7 | +import { fetchJson, installTarball, isCached, packageCache, probe, readVersionFile, rootLockHash, toolArch, toolDir } from "./tools.js"; | |
| 6 | 8 | |
| 7 | 9 | interface NodeDistEntry { |
| 8 | 10 | version: string; // "v22.11.0" |
| @@ -55,8 +57,7 @@ | ||
| 55 | 57 | ok = true; |
| 56 | 58 | } |
| 57 | 59 | } else { |
| 58 | - const want = parseVer(s); | |
| 59 | - ok = satisfies(installed, s) || (!!want && /^\d/.test(s) && want[0] === parseVer(installed)![0]); | |
| 60 | + ok = satisfies(installed, s); | |
| 60 | 61 | } |
| 61 | 62 | } |
| 62 | 63 | if (ok) { |
| @@ -83,43 +84,72 @@ | ||
| 83 | 84 | if (version) c.setOutput("node-version", `v${version}`); |
| 84 | 85 | |
| 85 | 86 | const registry = c.input("registry-url"); |
| 86 | - if (registry) { | |
| 87 | - // Minimal setup-node `registry-url` support: write an .npmrc that reads NODE_AUTH_TOKEN. | |
| 88 | - const { writeFile, mkdir } = await import("node:fs/promises"); | |
| 89 | - const npmrc = path.join(c.job.tempDir, ".npmrc"); | |
| 90 | - await mkdir(c.job.tempDir, { recursive: true }); | |
| 91 | - const url = registry.endsWith("/") ? registry : `${registry}/`; | |
| 92 | - const scope = c.input("scope"); | |
| 93 | - const authLine = `${url.replace(/^\w+:/, "")}:_authToken=\${NODE_AUTH_TOKEN}`; | |
| 94 | - const regLine = scope ? `${scope.startsWith("@") ? scope : "@" + scope}:registry=${url}` : `registry=${url}`; | |
| 95 | - await writeFile(npmrc, `${authLine}\n${regLine}\n`); | |
| 96 | - c.exportVariable("NPM_CONFIG_USERCONFIG", npmrc); | |
| 97 | - c.exportVariable("NODE_AUTH_TOKEN", "XXXXX-XXXXX-XXXXX-XXXXX"); | |
| 98 | - } | |
| 87 | + if (registry) await configureRegistry(c, registry); | |
| 99 | 88 | |
| 100 | - const pm = c.input("cache").toLowerCase(); | |
| 89 | + let pm = c.input("cache").toLowerCase(); | |
| 90 | + // setup-node v5+: caches npm automatically when package.json names npm as its package manager. | |
| 91 | + if (!pm && major(c) >= 5 && truthyInput(c.input("package-manager-cache"), true) && packageJsonUsesNpm(c.job.workspace)) { | |
| 92 | + c.log("Detected npm as the package manager from package.json's packageManager field. Auto caching has been enabled for npm. If you want to disable it, set package-manager-cache input to false"); | |
| 93 | + pm = "npm"; | |
| 94 | + } | |
| 101 | 95 | if (pm) { |
| 102 | 96 | const env = c.toolEnv(); |
| 103 | - let dir: string | null = null; | |
| 104 | - let locks: string[]; | |
| 97 | + let dir: string | null; | |
| 98 | + let lockFiles: string[]; | |
| 105 | 99 | if (pm === "npm") { |
| 106 | 100 | dir = await probe(c, "npm", ["config", "get", "cache"], env); |
| 107 | - locks = ["**/package-lock.json", "**/npm-shrinkwrap.json"]; | |
| 101 | + lockFiles = ["package-lock.json", "npm-shrinkwrap.json", "yarn.lock"]; | |
| 108 | 102 | } else if (pm === "pnpm") { |
| 109 | 103 | dir = await probe(c, "pnpm", ["store", "path", "--silent"], env); |
| 110 | - locks = ["**/pnpm-lock.yaml"]; | |
| 104 | + lockFiles = ["pnpm-lock.yaml"]; | |
| 111 | 105 | } else if (pm === "yarn") { |
| 112 | 106 | const ver = await probe(c, "yarn", ["--version"], env); |
| 113 | 107 | dir = ver && parseVer(ver)![0] >= 2 ? await probe(c, "yarn", ["config", "get", "cacheFolder"], env) : await probe(c, "yarn", ["cache", "dir"], env); |
| 114 | - locks = ["**/yarn.lock"]; | |
| 108 | + lockFiles = ["yarn.lock"]; | |
| 115 | 109 | } else { |
| 116 | 110 | throw failure(`Caching for '${pm}' is not supported`); |
| 117 | 111 | } |
| 118 | - await packageCache(c, { | |
| 119 | - prefix: `node-cache-Linux-${toolArch("node")}-${pm}`, | |
| 120 | - paths: dir ? [dir] : [], | |
| 121 | - lockPatterns: locks, | |
| 122 | - what: pm, | |
| 123 | - }); | |
| 112 | + const hash = rootLockHash(c, c.input("cache-dependency-path"), lockFiles); | |
| 113 | + if (!hash) throw failure("Some specified paths were not resolved, unable to cache dependencies."); | |
| 114 | + const key = `node-cache-Linux-${toolArch("node")}-${pm}-${hash}`; | |
| 115 | + c.setOutput("cache-primary-key", key); | |
| 116 | + await packageCache(c, { key, paths: dir ? [dir] : [], what: pm }); | |
| 124 | 117 | } |
| 125 | 118 | }; |
| 119 | + | |
| 120 | +function major(c: { uses: string }): number { | |
| 121 | + return Number(/@v?(\d+)/.exec(c.uses)?.[1] ?? 99); | |
| 122 | +} | |
| 123 | + | |
| 124 | +function packageJsonUsesNpm(workspace: string): boolean { | |
| 125 | + const npm = /^(\^)?npm(@.*)?$/; | |
| 126 | + try { | |
| 127 | + const pkg = JSON.parse(readFileSync(path.join(workspace, "package.json"), "utf8")) as { packageManager?: unknown; devEngines?: { packageManager?: unknown } }; | |
| 128 | + const dev = pkg.devEngines?.packageManager; | |
| 129 | + for (const o of Array.isArray(dev) ? dev : dev ? [dev] : []) if (typeof (o as { name?: unknown })?.name === "string" && npm.test((o as { name: string }).name)) return true; | |
| 130 | + return typeof pkg.packageManager === "string" && npm.test(pkg.packageManager); | |
| 131 | + } catch { | |
| 132 | + return false; | |
| 133 | + } | |
| 134 | +} | |
| 135 | + | |
| 136 | +/** setup-node's `registry-url`: an .npmrc in RUNNER_TEMP that reads NODE_AUTH_TOKEN. */ | |
| 137 | +async function configureRegistry(c: BuiltinContext, registry: string): Promise<void> { | |
| 138 | + const url = registry.endsWith("/") ? registry : `${registry}/`; | |
| 139 | + let scope = c.input("scope"); | |
| 140 | + if (!scope && url.includes("npm.pkg.github.com")) scope = c.job.spec.repo.owner; | |
| 141 | + if (scope && !scope.startsWith("@")) scope = `@${scope}`; | |
| 142 | + const prefix = scope ? `${scope.toLowerCase()}:` : ""; | |
| 143 | + const npmrc = path.join(c.job.tempDir, ".npmrc"); | |
| 144 | + await mkdir(c.job.tempDir, { recursive: true }); | |
| 145 | + let contents = ""; | |
| 146 | + if (existsSync(npmrc)) { | |
| 147 | + for (const line of (await readFile(npmrc, "utf8")).split("\n")) if (!line.toLowerCase().startsWith(`${prefix}registry`)) contents += `${line}\n`; | |
| 148 | + } | |
| 149 | + contents += `${url.replace(/^\w+:/, "")}:_authToken=\${NODE_AUTH_TOKEN}\n${prefix}registry=${url}`; | |
| 150 | + const alwaysAuth = c.input("always-auth"); | |
| 151 | + if (major(c) < 5 || alwaysAuth) contents += `\nalways-auth=${alwaysAuth || "false"}`; | |
| 152 | + await writeFile(npmrc, contents); | |
| 153 | + c.exportVariable("NPM_CONFIG_USERCONFIG", npmrc); | |
| 154 | + c.exportVariable("NODE_AUTH_TOKEN", c.toolEnv().NODE_AUTH_TOKEN || "XXXXX-XXXXX-XXXXX-XXXXX"); | |
| 155 | +} |
modifiedcontainers/runner/src/builtins/setup-python.ts+42−13View file
| @@ -1,10 +1,11 @@ | ||
| 1 | -import { existsSync, realpathSync } from "node:fs"; | |
| 1 | +import { existsSync, readFileSync, realpathSync } from "node:fs"; | |
| 2 | 2 | import { mkdir, rm, symlink } from "node:fs/promises"; |
| 3 | 3 | import { randomUUID } from "node:crypto"; |
| 4 | +import { homedir } from "node:os"; | |
| 4 | 5 | import path from "node:path"; |
| 5 | 6 | import { errMsg, failure } from "../job.js"; |
| 6 | -import { maxSatisfying, satisfies, which } from "../util.js"; | |
| 7 | -import type { Builtin } from "./index.js"; | |
| 7 | +import { hashFiles, inputList, maxSatisfying, satisfies, truthyInput, which } from "../util.js"; | |
| 8 | +import type { Builtin, BuiltinContext } from "./index.js"; | |
| 8 | 9 | import { fetchJson, fetchToFile, packageCache, probe, readVersionFile, toolArch } from "./tools.js"; |
| 9 | 10 | |
| 10 | 11 | interface PyManifestEntry { |
| @@ -53,8 +54,10 @@ | ||
| 53 | 54 | throw failure(`Unable to fetch the Python versions manifest: ${errMsg(e)}`); |
| 54 | 55 | } |
| 55 | 56 | const arch = toolArch("python"); |
| 56 | - const candidates = manifest.filter((m) => m.stable && m.files.some((f) => f.platform === "linux" && f.arch === arch)); | |
| 57 | - const v = maxSatisfying(candidates.map((m) => m.version), want); | |
| 57 | + const candidates = manifest.filter((m) => m.files.some((f) => f.platform === "linux" && f.arch === arch)); | |
| 58 | + let v = maxSatisfying(candidates.filter((m) => m.stable).map((m) => m.version), want); | |
| 59 | + // allow-prereleases (and `-dev` specs) fall back to pre-release builds when no stable one matches. | |
| 60 | + if (!v && (truthyInput(c.input("allow-prereleases"), false) || spec.endsWith("-dev"))) v = maxSatisfying(candidates.map((m) => m.version), want); | |
| 58 | 61 | if (!v) throw failure(`The version '${spec}' with architecture '${arch}' was not found for this operating system.`); |
| 59 | 62 | const entry = candidates.find((m) => m.version === v)!; |
| 60 | 63 | const files = entry.files.filter((f) => f.platform === "linux" && f.arch === arch); |
| @@ -89,13 +92,39 @@ | ||
| 89 | 92 | |
| 90 | 93 | const cache = c.input("cache").toLowerCase(); |
| 91 | 94 | if (cache) { |
| 92 | - if (cache !== "pip") throw failure(`Caching for '${cache}' is not supported by the Diffinery setup-python built-in (only pip).`); | |
| 93 | - const dir = await probe(c, "python3", ["-m", "pip", "cache", "dir"], c.toolEnv()); | |
| 94 | - await packageCache(c, { | |
| 95 | - prefix: `setup-python-Linux-${toolArch("python")}-python-${version}-pip`, | |
| 96 | - paths: dir ? [dir] : [], | |
| 97 | - lockPatterns: ["**/requirements*.txt", "**/pyproject.toml"], | |
| 98 | - what: "pip", | |
| 99 | - }); | |
| 95 | + const ws = c.job.workspace; | |
| 96 | + const arch = process.arch; | |
| 97 | + const deps = c.input("cache-dependency-path"); | |
| 98 | + if (cache === "pip") { | |
| 99 | + const hash = hashFiles(ws, inputList(deps || "**/requirements.txt")) || hashFiles(ws, ["**/pyproject.toml"]); | |
| 100 | + if (!hash) throw failure(`No file in ${ws} matched to [${inputList(deps || "**/requirements.txt").join(",")} or **/pyproject.toml], make sure you have checked out the target repository`); | |
| 101 | + let dir = await probe(c, "pip", ["cache", "dir"], c.toolEnv()); | |
| 102 | + if (dir?.includes("~")) dir = path.join(homedir(), dir.slice(dir.indexOf("~") + 1)); | |
| 103 | + const os = await linuxInfo(c); | |
| 104 | + const prefix = `setup-python-Linux-${arch}-${os.version}-${os.name}-python-${version}-pip`; | |
| 105 | + await packageCache(c, { key: `${prefix}-${hash}`, restoreKeys: [prefix], paths: dir ? [dir] : [], what: "pip" }); | |
| 106 | + } else if (cache === "pipenv") { | |
| 107 | + const hash = hashFiles(ws, inputList(deps || "**/Pipfile.lock")); | |
| 108 | + if (!hash) throw failure(`No file in ${ws} matched to [${inputList(deps || "**/Pipfile.lock").join(",")}], make sure you have checked out the target repository`); | |
| 109 | + await packageCache(c, { key: `setup-python-Linux-${arch}-python-${version}-pipenv-${hash}`, paths: [path.join(homedir(), ".local/share/virtualenvs")], what: "pipenv" }); | |
| 110 | + } else { | |
| 111 | + throw failure(`Caching for '${cache}' is not supported`); | |
| 112 | + } | |
| 100 | 113 | } |
| 101 | 114 | }; |
| 115 | + | |
| 116 | +/** setup-python's `lsb_release -i -r -s` (falls back to /etc/os-release). */ | |
| 117 | +async function linuxInfo(c: BuiltinContext): Promise<{ name: string; version: string }> { | |
| 118 | + const out = await probe(c, "lsb_release", ["-i", "-r", "-s"]); | |
| 119 | + if (out) { | |
| 120 | + const [name, version] = out.split("\n").map((s) => s.trim()); | |
| 121 | + if (name && version) return { name, version }; | |
| 122 | + } | |
| 123 | + try { | |
| 124 | + const rel = readFileSync("/etc/os-release", "utf8"); | |
| 125 | + const get = (k: string) => new RegExp(`^${k}="?([^"\n]*)"?`, "m").exec(rel)?.[1] ?? ""; | |
| 126 | + return { name: get("NAME").split(" ")[0] || "Linux", version: get("VERSION_ID") }; | |
| 127 | + } catch { | |
| 128 | + return { name: "Linux", version: "" }; | |
| 129 | + } | |
| 130 | +} |
modifiedcontainers/runner/src/builtins/tools.ts+24−15View file
| @@ -7,7 +7,7 @@ | ||
| 7 | 7 | import type { ReadableStream as WebReadableStream } from "node:stream/web"; |
| 8 | 8 | import { failure } from "../job.js"; |
| 9 | 9 | import { hashFiles, inputList } from "../util.js"; |
| 10 | -import { restoreCache, saveCache } from "./cache.js"; | |
| 10 | +import { isExactKeyMatch, restoreCache, saveCache } from "./cache.js"; | |
| 11 | 11 | import type { BuiltinContext } from "./index.js"; |
| 12 | 12 | |
| 13 | 13 | /** Downloads a public URL to a file with a few retries. */ |
| @@ -122,27 +122,36 @@ | ||
| 122 | 122 | } |
| 123 | 123 | |
| 124 | 124 | /** |
| 125 | - * Restores a package-manager cache keyed by lockfile hash and registers a | |
| 126 | - * post-step save, mirroring setup-node/setup-python/setup-go `cache:`. | |
| 125 | + * Restores a package-manager cache and registers a post-step save, the way | |
| 126 | + * setup-node/setup-python/setup-go `cache:` do (same keys and paths, so their | |
| 127 | + * caches are shared with the official actions). | |
| 127 | 128 | */ |
| 128 | -export async function packageCache(c: BuiltinContext, o: { prefix: string; paths: string[]; lockPatterns: string[]; what: string }): Promise<void> { | |
| 129 | - const patterns = inputList(c.input("cache-dependency-path")).length ? inputList(c.input("cache-dependency-path")) : o.lockPatterns; | |
| 130 | - const hash = hashFiles(c.job.workspace, patterns); | |
| 131 | - if (!hash) { | |
| 132 | - throw failure(`Dependencies lock file is not found in ${c.job.workspace}. Supported file patterns: ${patterns.join(", ")}`); | |
| 133 | - } | |
| 134 | - const key = `${o.prefix}-${hash}`; | |
| 129 | +export async function packageCache(c: BuiltinContext, o: { key: string; paths: string[]; restoreKeys?: string[]; what: string }): Promise<void> { | |
| 135 | 130 | const paths = o.paths.filter(Boolean); |
| 136 | 131 | if (!paths.length) { |
| 137 | 132 | c.warn(`Could not determine the ${o.what} cache directory; skipping cache.`); |
| 138 | 133 | return; |
| 139 | 134 | } |
| 140 | - const r = await restoreCache(c, paths, key, []); | |
| 141 | - c.setOutput("cache-hit", r.exact ? "true" : "false"); | |
| 142 | - if (r.exact) return; | |
| 135 | + const r = await restoreCache(c, paths, o.key, o.restoreKeys ?? []); | |
| 136 | + c.setOutput("cache-hit", isExactKeyMatch(o.key, r.matchedKey) ? "true" : "false"); | |
| 137 | + c.log(r.matchedKey ? `Cache restored from key: ${r.matchedKey}` : `${o.what} cache is not found`); | |
| 143 | 138 | c.registerPost(`Post ${c.name}`, "success()", async (pc) => { |
| 144 | - for (const p of paths) await mkdir(p, { recursive: true }).catch(() => {}); | |
| 145 | - await saveCache(pc, paths, key); | |
| 139 | + if (isExactKeyMatch(o.key, r.matchedKey)) { | |
| 140 | + pc.log(`Cache hit occurred on the primary key ${o.key}, not saving cache.`); | |
| 141 | + return; | |
| 142 | + } | |
| 143 | + if (!paths.some((p) => existsSync(p))) { | |
| 144 | + pc.log(`The ${o.what} cache directories do not exist; not saving cache.`); | |
| 145 | + return; | |
| 146 | + } | |
| 147 | + await saveCache(pc, paths, o.key); | |
| 146 | 148 | }); |
| 147 | 149 | } |
| 148 | 150 | |
| 151 | +/** The hash of the first lock file that exists in the workspace root (setup-node/setup-go's default). */ | |
| 152 | +export function rootLockHash(c: BuiltinContext, input: string, rootFiles: string[]): string { | |
| 153 | + if (input) return hashFiles(c.job.workspace, inputList(input)); | |
| 154 | + const found = rootFiles.find((f) => existsSync(path.join(c.job.workspace, f))); | |
| 155 | + if (!found) throw failure(`Dependencies lock file is not found in ${c.job.workspace}. Supported file patterns: ${rootFiles.join(",")}`); | |
| 156 | + return hashFiles(c.job.workspace, [found]); | |
| 157 | +} |
modifiedcontainers/runner/src/job.ts+28−3View file
| @@ -10,6 +10,7 @@ | ||
| 10 | 10 | import { Docker, type Wrap } from "./docker.js"; |
| 11 | 11 | import { JobLogger, OutputProcessor, type Annotation, type OutputSink } from "./log.js"; |
| 12 | 12 | import { Masker } from "./mask.js"; |
| 13 | +import { RuntimeClient } from "./runtime.js"; | |
| 13 | 14 | import { runAction } from "./steps/actions.js"; |
| 14 | 15 | import { runScript } from "./steps/run.js"; |
| 15 | 16 | import { exec, hashFiles, killTree, nowIso, runnerArch, type ExecResult } from "./util.js"; |
| @@ -64,7 +65,8 @@ | ||
| 64 | 65 | |
| 65 | 66 | export class StepFailure extends Error {} |
| 66 | 67 | |
| 67 | -const SENSITIVE_ENV = ["DIFFINERY_JOB_TOKEN", "DIFFINERY_SANDBOX_SECRET"]; | |
| 68 | +/** The runner's own credentials: never in a step's environment. */ | |
| 69 | +const SENSITIVE_ENV = ["DIFFINERY_JOB_TOKEN", "DIFFINERY_SANDBOX_SECRET", "DIFFINERY_GITHUB_TOKEN"]; | |
| 68 | 70 | |
| 69 | 71 | export class JobRunner { |
| 70 | 72 | readonly masker = new Masker(); |
| @@ -89,11 +91,19 @@ | ||
| 89 | 91 | readonly stepReasons = new Map<number, string>(); |
| 90 | 92 | /** Job and service containers, and Docker actions (self-hosted runners with Docker). */ |
| 91 | 93 | readonly docker: Docker; |
| 94 | + /** The Actions runtime services client (null when the server sent none). */ | |
| 95 | + readonly runtime: RuntimeClient | null; | |
| 92 | 96 | |
| 93 | 97 | private stopping: null | "cancelled" | "timeout" = null; |
| 94 | 98 | private firstFailure: { step?: number; message: string } | null = null; |
| 95 | 99 | private githubCtx: Record<string, unknown> = {}; |
| 96 | 100 | private defaultEnv: Record<string, string> = {}; |
| 101 | + /** | |
| 102 | + * The Actions runtime services' variables. Like GitHub, only JavaScript and | |
| 103 | + * Docker actions get them (and the built-ins, which call the services directly); | |
| 104 | + * `run:` steps do not. | |
| 105 | + */ | |
| 106 | + runtimeEnv: Record<string, string> = {}; | |
| 97 | 107 | private baseEnv: NodeJS.ProcessEnv; |
| 98 | 108 | private secrets: Record<string, string>; |
| 99 | 109 | |
| @@ -117,6 +127,8 @@ | ||
| 117 | 127 | for (const k of SENSITIVE_ENV) delete base[k]; |
| 118 | 128 | this.baseEnv = base; |
| 119 | 129 | for (const v of Object.values(this.secrets)) this.masker.add(v); |
| 130 | + if (spec.runtime) this.masker.add(spec.runtime.token); | |
| 131 | + this.runtime = RuntimeClient.fromSpec(spec); | |
| 120 | 132 | this.docker = new Docker(this); |
| 121 | 133 | } |
| 122 | 134 | |
| @@ -256,7 +268,7 @@ | ||
| 256 | 268 | GITHUB_REF_TYPE: refType, |
| 257 | 269 | GITHUB_HEAD_REF: str(g.head_ref), |
| 258 | 270 | GITHUB_BASE_REF: str(g.base_ref), |
| 259 | - GITHUB_RUN_ID: spec.runId, | |
| 271 | + GITHUB_RUN_ID: str(this.githubCtx.run_id), | |
| 260 | 272 | GITHUB_RUN_NUMBER: str(this.githubCtx.run_number), |
| 261 | 273 | GITHUB_RUN_ATTEMPT: str(this.githubCtx.run_attempt), |
| 262 | 274 | GITHUB_JOB: spec.jobKey, |
| @@ -279,8 +291,21 @@ | ||
| 279 | 291 | RUNNER_TOOL_CACHE: this.toolCache, |
| 280 | 292 | RUNNER_WORKSPACE: path.dirname(this.workspace), |
| 281 | 293 | AGENT_TOOLSDIRECTORY: this.toolCache, |
| 294 | + // What hosted ubuntu-24.04 runners report; setup-* actions key tool and dependency caches on it. | |
| 295 | + ImageOS: process.env.ImageOS ?? "ubuntu24", | |
| 296 | + ImageVersion: process.env.ImageVersion ?? process.env.DIFFINERY_IMAGE_VERSION ?? "20250101.1", | |
| 282 | 297 | }; |
| 283 | - if (this.debug) d.RUNNER_DEBUG = "1"; | |
| 298 | + if (this.debug || this.secrets.ACTIONS_RUNNER_DEBUG === "true" || spec.vars?.ACTIONS_RUNNER_DEBUG === "true") d.RUNNER_DEBUG = "1"; | |
| 299 | + if (spec.runtime) { | |
| 300 | + // The Actions runtime services (cache, artifacts), scoped to this job and expiring with it. | |
| 301 | + this.runtimeEnv = { | |
| 302 | + ACTIONS_RUNTIME_TOKEN: spec.runtime.token, | |
| 303 | + ACTIONS_RESULTS_URL: spec.runtime.resultsUrl, | |
| 304 | + ACTIONS_CACHE_URL: spec.runtime.cacheUrl, | |
| 305 | + ACTIONS_RUNTIME_URL: spec.runtime.runtimeUrl, | |
| 306 | + ACTIONS_CACHE_SERVICE_V2: "true", | |
| 307 | + }; | |
| 308 | + } | |
| 284 | 309 | if (spec.idToken && this.api.online) { |
| 285 | 310 | // @actions/core appends `&audience=...`, so the URL must already carry a query string. |
| 286 | 311 | d.ACTIONS_ID_TOKEN_REQUEST_URL = `${this.api.url("/oidc")}?api-version=2.0`; |
addedcontainers/runner/src/runtime.ts+151−0View file
| @@ -1,0 +1,151 @@ | ||
| 1 | +import { createWriteStream } from "node:fs"; | |
| 2 | +import { open, stat } from "node:fs/promises"; | |
| 3 | +import { Readable } from "node:stream"; | |
| 4 | +import { pipeline } from "node:stream/promises"; | |
| 5 | +import type { ReadableStream as WebReadableStream } from "node:stream/web"; | |
| 6 | +import type { JobSpec } from "@diffinery/shared"; | |
| 7 | + | |
| 8 | +/** | |
| 9 | + * Client for the Actions runtime services (ACTIONS_RESULTS_URL): the Twirp | |
| 10 | + * cache and artifact services and their signed blob URLs. The built-in cache | |
| 11 | + * and artifact actions use it, so they share storage (and archive formats) | |
| 12 | + * with the official actions. | |
| 13 | + */ | |
| 14 | + | |
| 15 | +export class RuntimeError extends Error { | |
| 16 | + constructor( | |
| 17 | + readonly status: number, | |
| 18 | + readonly code: string, | |
| 19 | + message: string, | |
| 20 | + ) { | |
| 21 | + super(message); | |
| 22 | + } | |
| 23 | +} | |
| 24 | + | |
| 25 | +const sleep = (ms: number) => new Promise((r) => setTimeout(r, ms)); | |
| 26 | +const retryable = (status: number) => status === 408 || status === 429 || status >= 500; | |
| 27 | + | |
| 28 | +/** Uploads at most this much in one request; larger files go up as blocks. */ | |
| 29 | +const SINGLE_SHOT = 32 * 1024 * 1024; | |
| 30 | +const BLOCK_SIZE = 32 * 1024 * 1024; | |
| 31 | +const UPLOAD_CONCURRENCY = 4; | |
| 32 | + | |
| 33 | +export class RuntimeClient { | |
| 34 | + constructor( | |
| 35 | + readonly token: string, | |
| 36 | + readonly resultsUrl: string, | |
| 37 | + private readonly opts: { retries?: number; minBackoffMs?: number } = {}, | |
| 38 | + ) {} | |
| 39 | + | |
| 40 | + static fromSpec(spec: JobSpec): RuntimeClient | null { | |
| 41 | + return spec.runtime ? new RuntimeClient(spec.runtime.token, spec.runtime.resultsUrl) : null; | |
| 42 | + } | |
| 43 | + | |
| 44 | + private async withRetries<T>(what: string, fn: () => Promise<T>): Promise<T> { | |
| 45 | + const retries = this.opts.retries ?? 5; | |
| 46 | + let last: unknown; | |
| 47 | + for (let attempt = 0; attempt <= retries; attempt++) { | |
| 48 | + if (attempt) await sleep((this.opts.minBackoffMs ?? 1000) * 2 ** (attempt - 1)); | |
| 49 | + try { | |
| 50 | + return await fn(); | |
| 51 | + } catch (e) { | |
| 52 | + if (e instanceof RuntimeError && !retryable(e.status)) throw e; | |
| 53 | + last = e; | |
| 54 | + } | |
| 55 | + } | |
| 56 | + throw last instanceof Error ? new Error(`${what}: ${last.message}`) : new Error(`${what} failed`); | |
| 57 | + } | |
| 58 | + | |
| 59 | + /** Calls a Twirp method (JSON). Errors become RuntimeError with the Twirp code. */ | |
| 60 | + async twirp<T>(service: "CacheService" | "ArtifactService", method: string, body: Record<string, unknown>): Promise<T> { | |
| 61 | + const url = `${new URL(this.resultsUrl).origin}/twirp/github.actions.results.api.v1.${service}/${method}`; | |
| 62 | + return this.withRetries(method, async () => { | |
| 63 | + const res = await fetch(url, { | |
| 64 | + method: "POST", | |
| 65 | + headers: { authorization: `Bearer ${this.token}`, "content-type": "application/json", "user-agent": "diffinery-runner" }, | |
| 66 | + body: JSON.stringify(body), | |
| 67 | + }); | |
| 68 | + const text = await res.text(); | |
| 69 | + if (!res.ok) { | |
| 70 | + let code = "unknown"; | |
| 71 | + let msg = text.slice(0, 300); | |
| 72 | + try { | |
| 73 | + const j = JSON.parse(text) as { code?: string; msg?: string }; | |
| 74 | + code = j.code ?? code; | |
| 75 | + msg = j.msg ?? msg; | |
| 76 | + } catch { | |
| 77 | + /* not Twirp JSON */ | |
| 78 | + } | |
| 79 | + throw new RuntimeError(res.status, code, `${method} failed (${res.status} ${code}): ${msg}`); | |
| 80 | + } | |
| 81 | + return JSON.parse(text) as T; | |
| 82 | + }); | |
| 83 | + } | |
| 84 | + | |
| 85 | + /** Uploads a file to a signed URL (Put Blob, or Put Block + Put Block List). */ | |
| 86 | + async upload(url: string, file: string, contentType = "application/octet-stream"): Promise<number> { | |
| 87 | + const size = (await stat(file)).size; | |
| 88 | + const fh = await open(file, "r"); | |
| 89 | + try { | |
| 90 | + const read = async (offset: number, length: number) => { | |
| 91 | + const buf = Buffer.alloc(length); | |
| 92 | + let got = 0; | |
| 93 | + while (got < length) { | |
| 94 | + const { bytesRead } = await fh.read(buf, got, length - got, offset + got); | |
| 95 | + if (bytesRead === 0) break; | |
| 96 | + got += bytesRead; | |
| 97 | + } | |
| 98 | + return buf.subarray(0, got); | |
| 99 | + }; | |
| 100 | + const put = (target: string, body: Buffer, headers: Record<string, string>) => | |
| 101 | + this.withRetries("upload", async () => { | |
| 102 | + const res = await fetch(target, { method: "PUT", headers: { "x-ms-version": "2025-01-05", ...headers }, body }); | |
| 103 | + if (res.status !== 201) { | |
| 104 | + const text = await res.text().catch(() => ""); | |
| 105 | + throw new RuntimeError(res.status, "upload", `upload failed (${res.status}): ${text.slice(0, 300)}`); | |
| 106 | + } | |
| 107 | + }); | |
| 108 | + if (size <= SINGLE_SHOT) { | |
| 109 | + await put(url, await read(0, size), { "x-ms-blob-type": "BlockBlob", "x-ms-blob-content-type": contentType, "content-type": "application/octet-stream" }); | |
| 110 | + return size; | |
| 111 | + } | |
| 112 | + const blocks: string[] = []; | |
| 113 | + for (let off = 0, i = 0; off < size; off += BLOCK_SIZE, i++) blocks.push(Buffer.from(`block-${String(i).padStart(8, "0")}`).toString("base64")); | |
| 114 | + let next = 0; | |
| 115 | + const worker = async () => { | |
| 116 | + for (let i = next++; i < blocks.length; i = next++) { | |
| 117 | + const body = await read(i * BLOCK_SIZE, Math.min(BLOCK_SIZE, size - i * BLOCK_SIZE)); | |
| 118 | + await put(`${url}&comp=block&blockid=${encodeURIComponent(blocks[i]!)}`, body, { "content-type": "application/octet-stream" }); | |
| 119 | + } | |
| 120 | + }; | |
| 121 | + await Promise.all(Array.from({ length: Math.min(UPLOAD_CONCURRENCY, blocks.length) }, worker)); | |
| 122 | + const xml = `<?xml version="1.0" encoding="utf-8"?><BlockList>${blocks.map((b) => `<Latest>${b}</Latest>`).join("")}</BlockList>`; | |
| 123 | + await put(`${url}&comp=blocklist`, Buffer.from(xml), { "x-ms-blob-content-type": contentType, "content-type": "application/xml" }); | |
| 124 | + return size; | |
| 125 | + } finally { | |
| 126 | + await fh.close(); | |
| 127 | + } | |
| 128 | + } | |
| 129 | + | |
| 130 | + /** Downloads a signed URL to a file. Returns the response's content type and file name. */ | |
| 131 | + async download(url: string, file: string): Promise<{ contentType: string; fileName: string | null }> { | |
| 132 | + return this.withRetries("download", async () => { | |
| 133 | + const res = await fetch(url, { headers: { "user-agent": "diffinery-runner" } }); | |
| 134 | + if (!res.ok || !res.body) { | |
| 135 | + const text = await res.text().catch(() => ""); | |
| 136 | + throw new RuntimeError(res.status, "download", `download failed (${res.status}): ${text.slice(0, 200)}`); | |
| 137 | + } | |
| 138 | + await pipeline(Readable.fromWeb(res.body as unknown as WebReadableStream), createWriteStream(file)); | |
| 139 | + const cd = res.headers.get("content-disposition") ?? ""; | |
| 140 | + const star = /filename\*\s*=\s*UTF-8''([^;\r\n]*)/i.exec(cd)?.[1]; | |
| 141 | + const plain = /(?<!\*)filename\s*=\s*"?([^;\r\n"]*)"?/i.exec(cd)?.[1]; | |
| 142 | + let fileName: string | null = null; | |
| 143 | + try { | |
| 144 | + fileName = star ? decodeURIComponent(star) : (plain ?? null); | |
| 145 | + } catch { | |
| 146 | + fileName = plain ?? null; | |
| 147 | + } | |
| 148 | + return { contentType: (res.headers.get("content-type") ?? "").split(";")[0]!.trim().toLowerCase(), fileName }; | |
| 149 | + }); | |
| 150 | + } | |
| 151 | +} |
modifiedcontainers/runner/src/steps/actions.ts+57−3View file
| @@ -76,7 +76,7 @@ | ||
| 76 | 76 | const url = tpl.replace("{owner}", owner).replace("{repo}", repo).replace("{ref}", encodeURIComponent(ref)); |
| 77 | 77 | job.logger.line(io.logIndex, `Download action repository '${owner}/${repo}@${ref}'`); |
| 78 | 78 | const headers: Record<string, string> = { "user-agent": "diffinery-runner" }; |
| 79 | - const ghToken = process.env.DIFFINERY_GITHUB_TOKEN; | |
| 79 | + const ghToken = GITHUB_TOKEN; | |
| 80 | 80 | if (ghToken && url.startsWith("https://codeload.github.com/")) headers.authorization = `token ${ghToken}`; |
| 81 | 81 | let lastErr: unknown; |
| 82 | 82 | for (let attempt = 0; attempt < 4; attempt++) { |
| @@ -84,6 +84,10 @@ | ||
| 84 | 84 | try { |
| 85 | 85 | const res = await fetch(url, { headers, redirect: "follow" }); |
| 86 | 86 | if (res.status === 404) throw failure(`Unable to resolve action '${owner}/${repo}@${ref}', repository or version not found`); |
| 87 | + if (res.status === 401 || res.status === 403) { | |
| 88 | + lastErr = new Error(`HTTP ${res.status} downloading ${url}`); | |
| 89 | + break; | |
| 90 | + } | |
| 87 | 91 | if (!res.ok) throw new Error(`HTTP ${res.status} downloading ${url}`); |
| 88 | 92 | const buf = Buffer.from(await res.arrayBuffer()); |
| 89 | 93 | const tmp = `${dest}.tmp-${process.pid}-${Date.now()}`; |
| @@ -103,9 +107,38 @@ | ||
| 103 | 107 | lastErr = e; |
| 104 | 108 | } |
| 105 | 109 | } |
| 110 | + // The tarball endpoint can be unreachable (proxies, rate limits) while git still works. | |
| 111 | + if (!process.env.DIFFINERY_ACTIONS_DOWNLOAD_URL && (await cloneAction(dest, owner, repo, ref, ghToken))) return dest; | |
| 106 | 112 | throw failure(`Failed to download action '${owner}/${repo}@${ref}': ${errMsg(lastErr)}`); |
| 107 | 113 | } |
| 108 | 114 | |
| 115 | +/** Fetches `owner/repo@ref` (branch, tag or commit SHA) from github.com with git; false when that fails. */ | |
| 116 | +async function cloneAction(dest: string, owner: string, repo: string, ref: string, token: string | undefined): Promise<boolean> { | |
| 117 | + if (!/^[\w.-]+$/.test(owner) || !/^[\w.-]+$/.test(repo) || ref.startsWith("-") || /[\s\\]|\.\.|[~^:?*[]/.test(ref)) return false; | |
| 118 | + const tmp = `${dest}.tmp-${process.pid}-${Date.now()}`; | |
| 119 | + // The header goes through the environment, not argv, where other processes could read it. | |
| 120 | + const n = Number(process.env.GIT_CONFIG_COUNT) || 0; | |
| 121 | + const auth: Record<string, string> = token | |
| 122 | + ? { GIT_CONFIG_COUNT: String(n + 1), [`GIT_CONFIG_KEY_${n}`]: "http.https://github.com/.extraheader", [`GIT_CONFIG_VALUE_${n}`]: `AUTHORIZATION: basic ${Buffer.from(`x-access-token:${token}`).toString("base64")}` } | |
| 123 | + : {}; | |
| 124 | + const git = (args: string[]) => exec("git", args, { cwd: tmp, capture: true, env: { ...process.env, GIT_TERMINAL_PROMPT: "0", ...auth } }); | |
| 125 | + try { | |
| 126 | + await mkdir(tmp, { recursive: true }); | |
| 127 | + for (const args of [["init", "-q"], ["fetch", "-q", "--depth", "1", `https://github.com/${owner}/${repo}`, ref], ["checkout", "-q", "FETCH_HEAD"]]) { | |
| 128 | + if ((await git(args)).exitCode !== 0) throw new Error(args[0]); | |
| 129 | + } | |
| 130 | + await rm(path.join(tmp, ".git"), { recursive: true, force: true }); | |
| 131 | + await rm(dest, { recursive: true, force: true }); | |
| 132 | + await mkdir(path.dirname(dest), { recursive: true }); | |
| 133 | + await rename(tmp, dest); | |
| 134 | + await writeFile(path.join(dest, ".diffinery-complete"), ""); | |
| 135 | + return true; | |
| 136 | + } catch { | |
| 137 | + await rm(tmp, { recursive: true, force: true }); | |
| 138 | + return false; | |
| 139 | + } | |
| 140 | +} | |
| 141 | + | |
| 109 | 142 | export function inputEnvName(name: string): string { |
| 110 | 143 | return `INPUT_${name.replace(/ /g, "_").toUpperCase()}`; |
| 111 | 144 | } |
| @@ -158,9 +191,10 @@ | ||
| 158 | 191 | } |
| 159 | 192 | |
| 160 | 193 | if (ref.kind === "remote") { |
| 161 | - const builtin = getBuiltin(ref.name, ref.ref); | |
| 194 | + const given = computeInputs(job, x, null); | |
| 195 | + const builtin = getBuiltin(ref.name, ref.ref, given, x.envCtx); | |
| 162 | 196 | if (builtin) { |
| 163 | - const inputs = computeInputs(job, x, null); | |
| 197 | + const inputs = given; | |
| 164 | 198 | logActionHeader(job, x.io, uses, inputs); |
| 165 | 199 | const c = makeBuiltinContext(job, x, inputs, uses); |
| 166 | 200 | await builtin(c); |
| @@ -243,12 +277,20 @@ | ||
| 243 | 277 | const inputEnv: Record<string, string> = {}; |
| 244 | 278 | for (const [k, v] of Object.entries(inputs)) inputEnv[inputEnvName(k)] = v; |
| 245 | 279 | const baseExtra: Record<string, string> = { |
| 280 | + ...job.runtimeEnv, | |
| 246 | 281 | ...inputEnv, |
| 247 | 282 | GITHUB_ACTION: x.step.id ?? "__" + (action.repository || "local").replace(/[^A-Za-z0-9]/g, "_"), |
| 248 | 283 | GITHUB_ACTION_PATH: action.path, |
| 249 | 284 | GITHUB_ACTION_REPOSITORY: action.repository, |
| 250 | 285 | GITHUB_ACTION_REF: action.ref, |
| 251 | 286 | }; |
| 287 | + // @actions/artifact refuses to run on anything that looks like GitHub Enterprise Server | |
| 288 | + // (any GITHUB_SERVER_URL but github.com), so the artifact actions see github.com; their | |
| 289 | + // API calls still go to GITHUB_API_URL and the runtime services, and the URLs they | |
| 290 | + // output are pointed back at this server. | |
| 291 | + const serverUrl = job.processEnv(x.envCtx).GITHUB_SERVER_URL ?? ""; | |
| 292 | + const artifactAction = ARTIFACT_ACTIONS.has(action.repository.toLowerCase()); | |
| 293 | + if (artifactAction) baseExtra.GITHUB_SERVER_URL = "https://github.com"; | |
| 252 | 294 | const state: Record<string, string> = {}; |
| 253 | 295 | const runScriptFile = async (file: string, io: StepIO, envCtx: Record<string, string>, extra: Record<string, string>, label: string) => { |
| 254 | 296 | const res = await job.runStepProcess([job.nodePath(), path.join(action.path, file)], { |
| @@ -305,9 +347,20 @@ | ||
| 305 | 347 | await runScriptFile(runs.main, mainIo, x.envCtx, { ...baseExtra, ...stateEnvOf(state) }, "main"); |
| 306 | 348 | } finally { |
| 307 | 349 | Object.assign(state, mainIo.state); |
| 350 | + const url = mainIo.outputs["artifact-url"]; | |
| 351 | + if (artifactAction && serverUrl && url?.startsWith("https://github.com/")) mainIo.outputs["artifact-url"] = `${serverUrl}/${url.slice("https://github.com/".length)}`; | |
| 308 | 352 | } |
| 309 | 353 | } |
| 310 | 354 | |
| 355 | +const ARTIFACT_ACTIONS = new Set(["actions/upload-artifact", "actions/download-artifact"]); | |
| 356 | + | |
| 357 | +/** | |
| 358 | + * The operator's github.com token for downloading actions (rate limits, private | |
| 359 | + * actions), read once when the runner starts. Steps never see it: job.ts strips | |
| 360 | + * it from every child environment. | |
| 361 | + */ | |
| 362 | +const GITHUB_TOKEN = process.env.DIFFINERY_GITHUB_TOKEN || undefined; | |
| 363 | + | |
| 311 | 364 | /** Docker container actions: `uses: docker://image`, or an action with `runs.using: docker`. */ |
| 312 | 365 | async function runDockerAction( |
| 313 | 366 | job: JobRunner, |
| @@ -328,6 +381,7 @@ | ||
| 328 | 381 | ): Promise<void> { |
| 329 | 382 | const image = await job.docker.prepareAction(x.io, d.image, d.dir); |
| 330 | 383 | const extra: Record<string, string> = { |
| 384 | + ...job.runtimeEnv, | |
| 331 | 385 | GITHUB_ACTION: x.step.id ?? "__" + (d.action.repository || "docker").replace(/[^A-Za-z0-9]/g, "_"), |
| 332 | 386 | GITHUB_ACTION_REPOSITORY: d.action.repository, |
| 333 | 387 | GITHUB_ACTION_REF: d.action.ref, |
addedcontainers/runner/src/zip.ts+307−0View file
| @@ -1,0 +1,307 @@ | ||
| 1 | +import { createHash } from "node:crypto"; | |
| 2 | +import { createReadStream, createWriteStream } from "node:fs"; | |
| 3 | +import { chmod, mkdir, open, stat, type FileHandle } from "node:fs/promises"; | |
| 4 | +import path from "node:path"; | |
| 5 | +import { Transform } from "node:stream"; | |
| 6 | +import { pipeline } from "node:stream/promises"; | |
| 7 | +import zlib from "node:zlib"; | |
| 8 | + | |
| 9 | +/** | |
| 10 | + * Minimal ZIP writer and reader (deflate or store, ZIP64 when needed), for | |
| 11 | + * artifacts in the format upload-artifact and download-artifact use. | |
| 12 | + */ | |
| 13 | + | |
| 14 | +const CRC_TABLE = (() => { | |
| 15 | + const t = new Uint32Array(256); | |
| 16 | + for (let n = 0; n < 256; n++) { | |
| 17 | + let c = n; | |
| 18 | + for (let k = 0; k < 8; k++) c = c & 1 ? 0xedb88320 ^ (c >>> 1) : c >>> 1; | |
| 19 | + t[n] = c >>> 0; | |
| 20 | + } | |
| 21 | + return t; | |
| 22 | +})(); | |
| 23 | + | |
| 24 | +export function crc32(buf: Uint8Array, prev = 0): number { | |
| 25 | + const native = (zlib as unknown as { crc32?: (d: Uint8Array, v?: number) => number }).crc32; | |
| 26 | + if (native) return native(buf, prev); | |
| 27 | + let c = (prev ^ 0xffffffff) >>> 0; | |
| 28 | + for (let i = 0; i < buf.length; i++) c = CRC_TABLE[(c ^ buf[i]!) & 0xff]! ^ (c >>> 8); | |
| 29 | + return (c ^ 0xffffffff) >>> 0; | |
| 30 | +} | |
| 31 | + | |
| 32 | +const MAX32 = 0xffffffff; | |
| 33 | +/** Files at least this large get ZIP64 sizes up front (deflate can grow incompressible data a little). */ | |
| 34 | +const ZIP64_THRESHOLD = 0xf0000000; | |
| 35 | + | |
| 36 | +function dosDateTime(d: Date): { date: number; time: number } { | |
| 37 | + const year = Math.max(1980, d.getFullYear()); | |
| 38 | + return { | |
| 39 | + date: ((year - 1980) << 9) | ((d.getMonth() + 1) << 5) | d.getDate(), | |
| 40 | + time: (d.getHours() << 11) | (d.getMinutes() << 5) | Math.floor(d.getSeconds() / 2), | |
| 41 | + }; | |
| 42 | +} | |
| 43 | + | |
| 44 | +export interface ZipEntry { | |
| 45 | + /** File to read. */ | |
| 46 | + source: string; | |
| 47 | + /** Path inside the archive, with `/` separators and no leading slash. */ | |
| 48 | + name: string; | |
| 49 | +} | |
| 50 | + | |
| 51 | +/** Writes `entries` to `out`. Returns the archive's size and SHA-256 (hex). */ | |
| 52 | +export async function writeZip(out: string, entries: ZipEntry[], level = 6): Promise<{ size: number; sha256: string }> { | |
| 53 | + const fh = await open(out, "w"); | |
| 54 | + let pos = 0; | |
| 55 | + const write = async (buf: Buffer) => { | |
| 56 | + await fh.write(buf, 0, buf.length, pos); | |
| 57 | + pos += buf.length; | |
| 58 | + }; | |
| 59 | + const central: Buffer[] = []; | |
| 60 | + try { | |
| 61 | + for (const e of entries) { | |
| 62 | + const st = await stat(e.source); | |
| 63 | + const name = Buffer.from(e.name.replace(/^\/+/, ""), "utf8"); | |
| 64 | + const { date, time } = dosDateTime(st.mtime); | |
| 65 | + const method = level === 0 ? 0 : 8; | |
| 66 | + const big = st.size >= ZIP64_THRESHOLD; | |
| 67 | + const offset = pos; | |
| 68 | + const local = Buffer.alloc(30); | |
| 69 | + local.writeUInt32LE(0x04034b50, 0); | |
| 70 | + local.writeUInt16LE(big ? 45 : 20, 4); | |
| 71 | + local.writeUInt16LE(0x0800, 6); // UTF-8 names | |
| 72 | + local.writeUInt16LE(method, 8); | |
| 73 | + local.writeUInt16LE(time, 10); | |
| 74 | + local.writeUInt16LE(date, 12); | |
| 75 | + local.writeUInt16LE(name.length, 26); | |
| 76 | + local.writeUInt16LE(big ? 20 : 0, 28); | |
| 77 | + await write(local); | |
| 78 | + await write(name); | |
| 79 | + const extraAt = pos; | |
| 80 | + if (big) await write(Buffer.alloc(20)); | |
| 81 | + let crc = 0; | |
| 82 | + let compressed = 0; | |
| 83 | + const counter = new Transform({ | |
| 84 | + transform(chunk: Buffer, _enc, cb) { | |
| 85 | + crc = crc32(chunk, crc); | |
| 86 | + cb(null, chunk); | |
| 87 | + }, | |
| 88 | + }); | |
| 89 | + const sink = async (source: AsyncIterable<Buffer>) => { | |
| 90 | + for await (const chunk of source) { | |
| 91 | + await write(chunk); | |
| 92 | + compressed += chunk.length; | |
| 93 | + } | |
| 94 | + }; | |
| 95 | + if (method === 8) await pipeline(createReadStream(e.source), counter, zlib.createDeflateRaw({ level }), sink); | |
| 96 | + else await pipeline(createReadStream(e.source), counter, sink); | |
| 97 | + // Patch sizes and CRC into the local header. | |
| 98 | + const fix = Buffer.alloc(12); | |
| 99 | + fix.writeUInt32LE(crc >>> 0, 0); | |
| 100 | + fix.writeUInt32LE(big ? MAX32 : compressed, 4); | |
| 101 | + fix.writeUInt32LE(big ? MAX32 : st.size, 8); | |
| 102 | + await fh.write(fix, 0, 12, offset + 14); | |
| 103 | + if (big) { | |
| 104 | + const z = Buffer.alloc(20); | |
| 105 | + z.writeUInt16LE(0x0001, 0); | |
| 106 | + z.writeUInt16LE(16, 2); | |
| 107 | + z.writeBigUInt64LE(BigInt(st.size), 4); | |
| 108 | + z.writeBigUInt64LE(BigInt(compressed), 12); | |
| 109 | + await fh.write(z, 0, 20, extraAt); | |
| 110 | + } else if (compressed > MAX32) { | |
| 111 | + throw new Error(`${e.name} does not fit in a 32-bit zip entry`); | |
| 112 | + } | |
| 113 | + // Central directory record, with ZIP64 fields for whatever overflows. | |
| 114 | + const z64: bigint[] = []; | |
| 115 | + if (big) z64.push(BigInt(st.size), BigInt(compressed)); | |
| 116 | + if (offset >= MAX32) z64.push(BigInt(offset)); | |
| 117 | + const extra = Buffer.alloc(z64.length ? 4 + 8 * z64.length : 0); | |
| 118 | + if (z64.length) { | |
| 119 | + extra.writeUInt16LE(0x0001, 0); | |
| 120 | + extra.writeUInt16LE(8 * z64.length, 2); | |
| 121 | + z64.forEach((v, i) => extra.writeBigUInt64LE(v, 4 + 8 * i)); | |
| 122 | + } | |
| 123 | + const cd = Buffer.alloc(46); | |
| 124 | + cd.writeUInt32LE(0x02014b50, 0); | |
| 125 | + cd.writeUInt16LE((3 << 8) | (z64.length ? 45 : 20), 4); // made by Unix | |
| 126 | + cd.writeUInt16LE(z64.length ? 45 : 20, 6); | |
| 127 | + cd.writeUInt16LE(0x0800, 8); | |
| 128 | + cd.writeUInt16LE(method, 10); | |
| 129 | + cd.writeUInt16LE(time, 12); | |
| 130 | + cd.writeUInt16LE(date, 14); | |
| 131 | + cd.writeUInt32LE(crc >>> 0, 16); | |
| 132 | + cd.writeUInt32LE(big ? MAX32 : compressed, 20); | |
| 133 | + cd.writeUInt32LE(big ? MAX32 : st.size, 24); | |
| 134 | + cd.writeUInt16LE(name.length, 28); | |
| 135 | + cd.writeUInt16LE(extra.length, 30); | |
| 136 | + cd.writeUInt32LE((((0o100000 | (st.mode & 0o777)) << 16) >>> 0) as number, 38); | |
| 137 | + cd.writeUInt32LE(offset >= MAX32 ? MAX32 : offset, 42); | |
| 138 | + central.push(cd, name, extra); | |
| 139 | + } | |
| 140 | + const cdStart = pos; | |
| 141 | + for (const b of central) await write(b); | |
| 142 | + const cdSize = pos - cdStart; | |
| 143 | + const zip64 = entries.length >= 0xffff || cdStart >= MAX32 || cdSize >= MAX32; | |
| 144 | + if (zip64) { | |
| 145 | + const eocd64At = pos; | |
| 146 | + const r = Buffer.alloc(56); | |
| 147 | + r.writeUInt32LE(0x06064b50, 0); | |
| 148 | + r.writeBigUInt64LE(44n, 4); | |
| 149 | + r.writeUInt16LE((3 << 8) | 45, 12); | |
| 150 | + r.writeUInt16LE(45, 14); | |
| 151 | + r.writeBigUInt64LE(BigInt(entries.length), 24); | |
| 152 | + r.writeBigUInt64LE(BigInt(entries.length), 32); | |
| 153 | + r.writeBigUInt64LE(BigInt(cdSize), 40); | |
| 154 | + r.writeBigUInt64LE(BigInt(cdStart), 48); | |
| 155 | + await write(r); | |
| 156 | + const loc = Buffer.alloc(20); | |
| 157 | + loc.writeUInt32LE(0x07064b50, 0); | |
| 158 | + loc.writeBigUInt64LE(BigInt(eocd64At), 8); | |
| 159 | + loc.writeUInt32LE(1, 16); | |
| 160 | + await write(loc); | |
| 161 | + } | |
| 162 | + const end = Buffer.alloc(22); | |
| 163 | + end.writeUInt32LE(0x06054b50, 0); | |
| 164 | + end.writeUInt16LE(Math.min(entries.length, 0xffff), 8); | |
| 165 | + end.writeUInt16LE(Math.min(entries.length, 0xffff), 10); | |
| 166 | + end.writeUInt32LE(Math.min(cdSize, MAX32), 12); | |
| 167 | + end.writeUInt32LE(Math.min(cdStart, MAX32), 16); | |
| 168 | + await write(end); | |
| 169 | + } finally { | |
| 170 | + await fh.close(); | |
| 171 | + } | |
| 172 | + return { size: pos, sha256: await sha256File(out) }; | |
| 173 | +} | |
| 174 | + | |
| 175 | +export async function sha256File(file: string): Promise<string> { | |
| 176 | + const h = createHash("sha256"); | |
| 177 | + await pipeline(createReadStream(file), h); | |
| 178 | + return h.digest("hex"); | |
| 179 | +} | |
| 180 | + | |
| 181 | +interface CentralEntry { | |
| 182 | + name: string; | |
| 183 | + method: number; | |
| 184 | + crc: number; | |
| 185 | + compressedSize: number; | |
| 186 | + size: number; | |
| 187 | + offset: number; | |
| 188 | + mode: number; | |
| 189 | +} | |
| 190 | + | |
| 191 | +async function readAt(fh: FileHandle, position: number, length: number): Promise<Buffer> { | |
| 192 | + const buf = Buffer.alloc(length); | |
| 193 | + const { bytesRead } = await fh.read(buf, 0, length, position); | |
| 194 | + return buf.subarray(0, bytesRead); | |
| 195 | +} | |
| 196 | + | |
| 197 | +async function centralDirectory(fh: FileHandle, fileSize: number): Promise<CentralEntry[]> { | |
| 198 | + const tailLen = Math.min(fileSize, 22 + 0xffff); | |
| 199 | + const tail = await readAt(fh, fileSize - tailLen, tailLen); | |
| 200 | + let at = -1; | |
| 201 | + for (let i = tail.length - 22; i >= 0; i--) { | |
| 202 | + if (tail.readUInt32LE(i) === 0x06054b50) { | |
| 203 | + at = i; | |
| 204 | + break; | |
| 205 | + } | |
| 206 | + } | |
| 207 | + if (at < 0) throw new Error("not a zip file (no end of central directory)"); | |
| 208 | + let count = tail.readUInt16LE(at + 10); | |
| 209 | + let cdSize = tail.readUInt32LE(at + 12); | |
| 210 | + let cdStart = tail.readUInt32LE(at + 16); | |
| 211 | + const eocdPos = fileSize - tailLen + at; | |
| 212 | + if (eocdPos >= 20) { | |
| 213 | + const loc = await readAt(fh, eocdPos - 20, 20); | |
| 214 | + if (loc.readUInt32LE(0) === 0x07064b50) { | |
| 215 | + const r = await readAt(fh, Number(loc.readBigUInt64LE(8)), 56); | |
| 216 | + if (r.readUInt32LE(0) !== 0x06064b50) throw new Error("corrupt zip64 end of central directory"); | |
| 217 | + count = Number(r.readBigUInt64LE(32)); | |
| 218 | + cdSize = Number(r.readBigUInt64LE(40)); | |
| 219 | + cdStart = Number(r.readBigUInt64LE(48)); | |
| 220 | + } | |
| 221 | + } | |
| 222 | + const cd = await readAt(fh, cdStart, cdSize); | |
| 223 | + const out: CentralEntry[] = []; | |
| 224 | + let p = 0; | |
| 225 | + for (let i = 0; i < count; i++) { | |
| 226 | + if (cd.readUInt32LE(p) !== 0x02014b50) throw new Error("corrupt zip central directory"); | |
| 227 | + const nameLen = cd.readUInt16LE(p + 28); | |
| 228 | + const extraLen = cd.readUInt16LE(p + 30); | |
| 229 | + const commentLen = cd.readUInt16LE(p + 32); | |
| 230 | + const e: CentralEntry = { | |
| 231 | + name: cd.subarray(p + 46, p + 46 + nameLen).toString("utf8"), | |
| 232 | + method: cd.readUInt16LE(p + 10), | |
| 233 | + crc: cd.readUInt32LE(p + 16), | |
| 234 | + compressedSize: cd.readUInt32LE(p + 20), | |
| 235 | + size: cd.readUInt32LE(p + 24), | |
| 236 | + offset: cd.readUInt32LE(p + 42), | |
| 237 | + mode: cd.readUInt8(p + 5) === 3 ? (cd.readUInt32LE(p + 38) >>> 16) & 0o777 : 0, | |
| 238 | + }; | |
| 239 | + // ZIP64 extra: the fields that are 0xffffffff, in order. | |
| 240 | + let x = p + 46 + nameLen; | |
| 241 | + const xEnd = x + extraLen; | |
| 242 | + while (x + 4 <= xEnd) { | |
| 243 | + const id = cd.readUInt16LE(x); | |
| 244 | + const len = cd.readUInt16LE(x + 2); | |
| 245 | + if (id === 0x0001) { | |
| 246 | + let q = x + 4; | |
| 247 | + if (e.size === MAX32) (e.size = Number(cd.readBigUInt64LE(q))), (q += 8); | |
| 248 | + if (e.compressedSize === MAX32) (e.compressedSize = Number(cd.readBigUInt64LE(q))), (q += 8); | |
| 249 | + if (e.offset === MAX32) e.offset = Number(cd.readBigUInt64LE(q)); | |
| 250 | + } | |
| 251 | + x += 4 + len; | |
| 252 | + } | |
| 253 | + out.push(e); | |
| 254 | + p += 46 + nameLen + extraLen + commentLen; | |
| 255 | + } | |
| 256 | + return out; | |
| 257 | +} | |
| 258 | + | |
| 259 | +/** Entry names that would escape the destination are refused. */ | |
| 260 | +export function safeEntryPath(dest: string, name: string): string { | |
| 261 | + const clean = name.replace(/\\/g, "/"); | |
| 262 | + if (clean.startsWith("/") || /^[A-Za-z]:/.test(clean) || clean.split("/").includes("..")) throw new Error(`refusing unsafe path in archive: ${name}`); | |
| 263 | + const target = path.resolve(dest, clean); | |
| 264 | + if (target !== path.resolve(dest) && !target.startsWith(path.resolve(dest) + path.sep)) throw new Error(`refusing unsafe path in archive: ${name}`); | |
| 265 | + return target; | |
| 266 | +} | |
| 267 | + | |
| 268 | +/** Extracts every entry of `zipFile` under `dest` (overwriting). Returns the number of files written. */ | |
| 269 | +export async function extractZip(zipFile: string, dest: string): Promise<number> { | |
| 270 | + const fh = await open(zipFile, "r"); | |
| 271 | + let files = 0; | |
| 272 | + try { | |
| 273 | + const entries = await centralDirectory(fh, (await fh.stat()).size); | |
| 274 | + await mkdir(dest, { recursive: true }); | |
| 275 | + for (const e of entries) { | |
| 276 | + const target = safeEntryPath(dest, e.name); | |
| 277 | + if (e.name.endsWith("/")) { | |
| 278 | + await mkdir(target, { recursive: true }); | |
| 279 | + continue; | |
| 280 | + } | |
| 281 | + if (e.method !== 0 && e.method !== 8) throw new Error(`unsupported compression method ${e.method} for ${e.name}`); | |
| 282 | + const local = await readAt(fh, e.offset, 30); | |
| 283 | + if (local.readUInt32LE(0) !== 0x04034b50) throw new Error(`corrupt local header for ${e.name}`); | |
| 284 | + const start = e.offset + 30 + local.readUInt16LE(26) + local.readUInt16LE(28); | |
| 285 | + await mkdir(path.dirname(target), { recursive: true }); | |
| 286 | + let crc = 0; | |
| 287 | + const check = new Transform({ | |
| 288 | + transform(chunk: Buffer, _enc, cb) { | |
| 289 | + crc = crc32(chunk, crc); | |
| 290 | + cb(null, chunk); | |
| 291 | + }, | |
| 292 | + }); | |
| 293 | + const raw = e.compressedSize ? createReadStream(zipFile, { start, end: start + e.compressedSize - 1 }) : null; | |
| 294 | + const out = createWriteStream(target); | |
| 295 | + if (!raw) out.end(); | |
| 296 | + else if (e.method === 8) await pipeline(raw, zlib.createInflateRaw(), check, out); | |
| 297 | + else await pipeline(raw, check, out); | |
| 298 | + if (raw && crc >>> 0 !== e.crc >>> 0) throw new Error(`CRC mismatch for ${e.name}`); | |
| 299 | + if (!raw) await new Promise<void>((r) => out.once("close", () => r())); | |
| 300 | + if (e.mode & 0o111) await chmod(target, e.mode).catch(() => {}); | |
| 301 | + files++; | |
| 302 | + } | |
| 303 | + } finally { | |
| 304 | + await fh.close(); | |
| 305 | + } | |
| 306 | + return files; | |
| 307 | +} |
modifiedcontainers/runner/test/helpers.ts+101−36View file
| @@ -1,5 +1,5 @@ | ||
| 1 | 1 | import { mkdtempSync } from "node:fs"; |
| 2 | -import { createServer, type Server } from "node:http"; | |
| 2 | +import { createServer, type IncomingHttpHeaders, type Server, type ServerResponse } from "node:http"; | |
| 3 | 3 | import type { AddressInfo } from "node:net"; |
| 4 | 4 | import { tmpdir } from "node:os"; |
| 5 | 5 | import path from "node:path"; |
| @@ -11,8 +11,13 @@ | ||
| 11 | 11 | logs: LogLine[]; |
| 12 | 12 | steps: Array<{ index: number; report: StepReport }>; |
| 13 | 13 | completion: JobCompletion | null; |
| 14 | + /** Runtime token the fake results service accepts (run "1", job "job-1"). */ | |
| 15 | + runtimeToken: string; | |
| 16 | + /** Cache entries by key (version in `cacheVersions`). */ | |
| 14 | 17 | cache: Map<string, Buffer>; |
| 15 | - artifacts: Map<string, Buffer>; | |
| 18 | + cacheVersions: Map<string, string>; | |
| 19 | + /** Finalized artifacts of the run, by name. */ | |
| 20 | + artifacts: Map<string, { id: number; body: Buffer; mimeType: string; hash?: string }>; | |
| 16 | 21 | requests: Array<{ method: string; path: string; auth: string | undefined }>; |
| 17 | 22 | /** Fail the next N log POSTs with 503 (to test retries). */ |
| 18 | 23 | failLogs: number; |
| @@ -29,7 +34,9 @@ | ||
| 29 | 34 | logs: [], |
| 30 | 35 | steps: [], |
| 31 | 36 | completion: null, |
| 37 | + runtimeToken: fakeRuntimeToken(), | |
| 32 | 38 | cache: new Map(), |
| 39 | + cacheVersions: new Map(), | |
| 33 | 40 | artifacts: new Map(), |
| 34 | 41 | requests: [], |
| 35 | 42 | failLogs: 0, |
| @@ -45,6 +52,90 @@ | ||
| 45 | 52 | return [...this.steps].reverse().find((s) => s.index === index)?.report; |
| 46 | 53 | }, |
| 47 | 54 | }; |
| 55 | + // A minimal Actions results service: the Twirp cache/artifact services and their blob URLs. | |
| 56 | + const pending = new Map<string, { kind: "cache" | "artifact"; name: string; version?: string; mimeType?: string; body?: Buffer }>(); | |
| 57 | + let nextId = 1; | |
| 58 | + const runtime = (method: string, u: URL, headers: IncomingHttpHeaders, body: Buffer, res: ServerResponse, json: (s: number, v: unknown) => void) => { | |
| 59 | + const blob = /^\/_runtime\/blob\/([^/]+)$/.exec(u.pathname); | |
| 60 | + if (blob) { | |
| 61 | + const id = decodeURIComponent(blob[1]!); | |
| 62 | + if (method === "PUT") { | |
| 63 | + const p = pending.get(id); | |
| 64 | + if (!p || headers["x-ms-blob-type"] !== "BlockBlob") return json(400, {}); | |
| 65 | + p.body = body; | |
| 66 | + res.writeHead(201); | |
| 67 | + return res.end(); | |
| 68 | + } | |
| 69 | + const art = [...api.artifacts.values()].find((a) => String(a.id) === id); | |
| 70 | + const b = art?.body ?? api.cache.get(id); | |
| 71 | + if (!b) return json(404, {}); | |
| 72 | + res.writeHead(200, { "content-type": art?.mimeType ?? "application/octet-stream" }); | |
| 73 | + return res.end(b); | |
| 74 | + } | |
| 75 | + if (headers.authorization !== `Bearer ${api.runtimeToken}`) return json(401, { code: "unauthenticated", msg: "bad token" }); | |
| 76 | + const req = JSON.parse(body.toString() || "{}") as Record<string, string | string[] | undefined>; | |
| 77 | + const blobUrl = (id: string) => `${api.url}/_runtime/blob/${encodeURIComponent(id)}?sig=x`; | |
| 78 | + switch (u.pathname.replace("/twirp/github.actions.results.api.v1.", "")) { | |
| 79 | + case "CacheService/GetCacheEntryDownloadURL": { | |
| 80 | + const keys = [...api.cache.keys()].filter((k) => api.cacheVersions.get(k) === req.version); | |
| 81 | + const hit = keys.find((k) => k === req.key) ?? ((req.restore_keys ?? []) as string[]).map((p) => keys.find((k) => k.startsWith(p))).find(Boolean); | |
| 82 | + return json(200, hit ? { ok: true, matched_key: hit, signed_download_url: blobUrl(hit) } : { ok: false }); | |
| 83 | + } | |
| 84 | + case "CacheService/CreateCacheEntry": { | |
| 85 | + if (api.cache.has(String(req.key))) return json(409, { code: "already_exists", msg: "cache entry already exists" }); | |
| 86 | + const id = `upload-${nextId++}`; | |
| 87 | + pending.set(id, { kind: "cache", name: String(req.key), version: String(req.version) }); | |
| 88 | + return json(200, { ok: true, signed_upload_url: blobUrl(id) }); | |
| 89 | + } | |
| 90 | + case "CacheService/FinalizeCacheEntryUpload": { | |
| 91 | + const [id, p] = [...pending].find(([, p]) => p.kind === "cache" && p.name === req.key) ?? []; | |
| 92 | + if (!id || !p?.body) return json(404, { code: "not_found", msg: "no upload" }); | |
| 93 | + pending.delete(id); | |
| 94 | + api.cache.set(p.name, p.body); | |
| 95 | + api.cacheVersions.set(p.name, p.version!); | |
| 96 | + return json(200, { ok: true, entry_id: String(nextId++) }); | |
| 97 | + } | |
| 98 | + case "ArtifactService/CreateArtifact": { | |
| 99 | + if (api.artifacts.has(String(req.name))) return json(409, { code: "already_exists", msg: "artifact exists" }); | |
| 100 | + const id = `upload-${nextId++}`; | |
| 101 | + pending.set(id, { kind: "artifact", name: String(req.name), mimeType: String(req.mime_type ?? "application/zip") }); | |
| 102 | + return json(200, { ok: true, signed_upload_url: blobUrl(id) }); | |
| 103 | + } | |
| 104 | + case "ArtifactService/FinalizeArtifact": { | |
| 105 | + const [id, p] = [...pending].find(([, p]) => p.kind === "artifact" && p.name === req.name) ?? []; | |
| 106 | + if (!id || !p?.body) return json(404, { code: "not_found", msg: "no upload" }); | |
| 107 | + pending.delete(id); | |
| 108 | + const artifactId = nextId++; | |
| 109 | + api.artifacts.set(p.name, { id: artifactId, body: p.body, mimeType: p.mimeType!, hash: req.hash as string | undefined }); | |
| 110 | + return json(200, { ok: true, artifact_id: String(artifactId) }); | |
| 111 | + } | |
| 112 | + case "ArtifactService/ListArtifacts": | |
| 113 | + return json(200, { | |
| 114 | + artifacts: [...api.artifacts] | |
| 115 | + .filter(([n]) => !req.name_filter || n === req.name_filter) | |
| 116 | + .map(([name, a]) => ({ | |
| 117 | + workflow_run_backend_id: "1", | |
| 118 | + workflow_job_run_backend_id: "job-1", | |
| 119 | + database_id: String(a.id), | |
| 120 | + name, | |
| 121 | + size: String(a.body.length), | |
| 122 | + digest: a.hash?.replace(/^sha256:/, ""), | |
| 123 | + created_at: new Date().toISOString(), | |
| 124 | + })), | |
| 125 | + }); | |
| 126 | + case "ArtifactService/GetSignedArtifactURL": { | |
| 127 | + const a = api.artifacts.get(String(req.name)); | |
| 128 | + return a ? json(200, { signed_url: blobUrl(String(a.id)) }) : json(404, { code: "not_found", msg: "no artifact" }); | |
| 129 | + } | |
| 130 | + case "ArtifactService/DeleteArtifact": { | |
| 131 | + const a = api.artifacts.get(String(req.name)); | |
| 132 | + if (!a) return json(404, { code: "not_found", msg: "no artifact" }); | |
| 133 | + api.artifacts.delete(String(req.name)); | |
| 134 | + return json(200, { ok: true, artifact_id: String(a.id) }); | |
| 135 | + } | |
| 136 | + } | |
| 137 | + return json(404, { code: "bad_route", msg: "no such method" }); | |
| 138 | + }; | |
| 48 | 139 | const server: Server = createServer(async (req, res) => { |
| 49 | 140 | const chunks: Buffer[] = []; |
| 50 | 141 | for await (const c of req) chunks.push(c as Buffer); |
| @@ -55,6 +146,7 @@ | ||
| 55 | 146 | res.writeHead(status, { "content-type": "application/json" }); |
| 56 | 147 | res.end(JSON.stringify(v)); |
| 57 | 148 | }; |
| 149 | + if (u.pathname.startsWith("/twirp/") || u.pathname.startsWith("/_runtime/blob/")) return runtime(req.method!, u, req.headers, body, res, json); | |
| 58 | 150 | if (req.headers.authorization !== `Bearer ${token}`) return json(401, { error: "unauthorized" }); |
| 59 | 151 | const m = /^\/runner\/jobs\/([^/]+)(\/.*)?$/.exec(u.pathname); |
| 60 | 152 | if (!m) return json(404, {}); |
| @@ -80,40 +172,6 @@ | ||
| 80 | 172 | api.completion = JSON.parse(body.toString()) as JobCompletion; |
| 81 | 173 | return json(200, {}); |
| 82 | 174 | } |
| 83 | - if (sub === "/cache" && req.method === "GET") { | |
| 84 | - const key = u.searchParams.get("key")!; | |
| 85 | - if (api.cache.has(key)) return json(200, { key, url: `${m[0].replace(/\/cache$/, "")}/cache/${encodeURIComponent(key)}` }); | |
| 86 | - for (const prefix of (u.searchParams.get("restore") ?? "").split(",").filter(Boolean)) { | |
| 87 | - const hit = [...api.cache.keys()].find((k) => k.startsWith(prefix)); | |
| 88 | - if (hit) return json(200, { key: hit, url: `/runner/jobs/${m[1]}/cache/${encodeURIComponent(hit)}` }); | |
| 89 | - } | |
| 90 | - return json(404, {}); | |
| 91 | - } | |
| 92 | - const cm = /^\/cache\/(.+)$/.exec(sub); | |
| 93 | - if (cm) { | |
| 94 | - const key = decodeURIComponent(cm[1]!); | |
| 95 | - if (req.method === "PUT") { | |
| 96 | - if (api.cache.has(key)) return json(409, {}); | |
| 97 | - api.cache.set(key, body); | |
| 98 | - return json(201, {}); | |
| 99 | - } | |
| 100 | - const b = api.cache.get(key); | |
| 101 | - if (!b) return json(404, {}); | |
| 102 | - res.writeHead(200, { "content-type": "application/gzip" }); | |
| 103 | - return res.end(b); | |
| 104 | - } | |
| 105 | - const am = /^\/artifacts\/(.+)$/.exec(sub); | |
| 106 | - if (am) { | |
| 107 | - const name = decodeURIComponent(am[1]!); | |
| 108 | - if (req.method === "PUT") { | |
| 109 | - api.artifacts.set(name, body); | |
| 110 | - return json(201, { id: name.length, url: `https://example.test/artifacts/${name}` }); | |
| 111 | - } | |
| 112 | - const b = api.artifacts.get(name); | |
| 113 | - if (!b) return json(404, {}); | |
| 114 | - res.writeHead(200, { "content-type": "application/gzip" }); | |
| 115 | - return res.end(b); | |
| 116 | - } | |
| 117 | 175 | if (sub === "/oidc") return json(200, { value: `oidc-for-${u.searchParams.get("audience")}` }); |
| 118 | 176 | return json(404, {}); |
| 119 | 177 | }); |
| @@ -123,6 +181,12 @@ | ||
| 123 | 181 | return api; |
| 124 | 182 | } |
| 125 | 183 | |
| 184 | +/** A runtime token shaped like the server's: its scope names run "1" and job "job-1". */ | |
| 185 | +function fakeRuntimeToken(): string { | |
| 186 | + const b64 = (v: unknown) => Buffer.from(JSON.stringify(v)).toString("base64url"); | |
| 187 | + return `${b64({ alg: "HS256", typ: "JWT" })}.${b64({ scp: "Actions.GenericRead:1 Actions.Results:1:job-1", exp: Math.floor(Date.now() / 1000) + 3600 })}.sig`; | |
| 188 | +} | |
| 189 | + | |
| 126 | 190 | export function tmp(prefix = "runner-test-"): string { |
| 127 | 191 | return mkdtempSync(path.join(tmpdir(), prefix)); |
| 128 | 192 | } |
| @@ -152,6 +216,7 @@ | ||
| 152 | 216 | idToken: false, |
| 153 | 217 | apiUrl: api.url, |
| 154 | 218 | token: api.token, |
| 219 | + runtime: { token: api.runtimeToken, resultsUrl: `${api.url}/`, cacheUrl: `${api.url}/_runtime/`, runtimeUrl: `${api.url}/_runtime/` }, | |
| 155 | 220 | ...over, |
| 156 | 221 | }; |
| 157 | 222 | } |
modifiedcontainers/runner/test/job.test.ts+193−7View file
| @@ -419,9 +419,53 @@ | ||
| 419 | 419 | } |
| 420 | 420 | }); |
| 421 | 421 | |
| 422 | - it("rejects github-script with a clear error", async () => { | |
| 423 | - const c2 = await run(makeSpec(api, [{ uses: "actions/github-script@v7", with: { script: "1" } }])); | |
| 424 | - expect(c2.failure?.message).toContain("actions/github-script is not supported"); | |
| 422 | + it("runs github-script, and built-ins it cannot mirror, as the real actions", async () => { | |
| 423 | + // fake codeload serving one node action for every repository | |
| 424 | + const src = tmp("real-action-"); | |
| 425 | + mkdirSync(path.join(src, "top"), { recursive: true }); | |
| 426 | + writeFileSync(path.join(src, "top/action.yml"), "name: real\nruns:\n using: node20\n main: main.js\n"); | |
| 427 | + writeFileSync( | |
| 428 | + path.join(src, "top/main.js"), | |
| 429 | + 'const e = process.env; console.log(`real ${e.GITHUB_ACTION_REPOSITORY}@${e.GITHUB_ACTION_REF} server=${e.GITHUB_SERVER_URL} results=${e.ACTIONS_RESULTS_URL} v2=${e.ACTIONS_CACHE_SERVICE_V2} token=${e.ACTIONS_RUNTIME_TOKEN ? "set" : "unset"} image=${e.ImageOS}`); require("fs").appendFileSync(e.GITHUB_OUTPUT, "artifact-url=https://github.com/acme/widgets/actions/runs/1/artifacts/7\\n");', | |
| 430 | + ); | |
| 431 | + const tarball = path.join(src, "a.tgz"); | |
| 432 | + execFileSync("tar", ["-czf", tarball, "-C", src, "top"]); | |
| 433 | + const hits: string[] = []; | |
| 434 | + const server = createServer((req, res) => { | |
| 435 | + hits.push(req.url ?? ""); | |
| 436 | + res.writeHead(200, { "content-type": "application/gzip" }); | |
| 437 | + res.end(readFileSync(tarball)); | |
| 438 | + }); | |
| 439 | + await new Promise<void>((r) => server.listen(0, "127.0.0.1", () => r())); | |
| 440 | + const prev = process.env.DIFFINERY_ACTIONS_DOWNLOAD_URL; | |
| 441 | + process.env.DIFFINERY_ACTIONS_DOWNLOAD_URL = `http://127.0.0.1:${(server.address() as AddressInfo).port}/{owner}/{repo}/{ref}`; | |
| 442 | + try { | |
| 443 | + const c = await run( | |
| 444 | + makeSpec(api, [ | |
| 445 | + { uses: "actions/github-script@v7", with: { script: "return 1" } }, | |
| 446 | + { uses: "actions/cache@v4", env: { DIFFINERY_BUILTINS: "false" }, with: { path: "x", key: "k" } }, | |
| 447 | + { uses: "actions/setup-node@v4", with: { "node-version": "20", "some-future-input": "x" } }, | |
| 448 | + { uses: "actions/checkout@main" }, | |
| 449 | + { id: "up", uses: "actions/upload-artifact@v4", with: { name: "a", path: "x", "unknown-input": "1" } }, | |
| 450 | + { run: 'echo "url=${{ steps.up.outputs.artifact-url }}"' }, | |
| 451 | + ]), | |
| 452 | + ); | |
| 453 | + expect(c.conclusion, api.logText()).toBe("success"); | |
| 454 | + expect(api.logText(0)).toContain(`real actions/github-script@v7 server=${api.url} results=${api.url}/ v2=true token=set image=ubuntu24`); | |
| 455 | + expect(api.logText(0)).not.toContain(api.runtimeToken); | |
| 456 | + expect(api.logText(1)).toContain("real actions/cache@v4"); | |
| 457 | + expect(api.logText(2)).toContain("real actions/setup-node@v4"); | |
| 458 | + expect(api.logText(3)).toContain("real actions/checkout@main"); | |
| 459 | + // @actions/artifact refuses GHES, so the artifact actions see github.com and their URLs are mapped back | |
| 460 | + expect(api.logText(4)).toContain("real actions/upload-artifact@v4 server=https://github.com"); | |
| 461 | + expect(api.logText(5)).toContain(`url=${api.url}/acme/widgets/actions/runs/1/artifacts/7`); | |
| 462 | + expect(hits).toEqual(["/actions/github-script/v7", "/actions/cache/v4", "/actions/setup-node/v4", "/actions/checkout/main", "/actions/upload-artifact/v4"]); | |
| 463 | + } finally { | |
| 464 | + if (prev === undefined) delete process.env.DIFFINERY_ACTIONS_DOWNLOAD_URL; | |
| 465 | + else process.env.DIFFINERY_ACTIONS_DOWNLOAD_URL = prev; | |
| 466 | + await new Promise<void>((r) => server.close(() => r())); | |
| 467 | + rmSync(src, { recursive: true, force: true }); | |
| 468 | + } | |
| 425 | 469 | }); |
| 426 | 470 | |
| 427 | 471 | it("enforces step timeout-minutes", async () => { |
| @@ -457,7 +501,7 @@ | ||
| 457 | 501 | expect(nums).toEqual(Array.from({ length: 500 }, (_, i) => `n${i + 1}`)); |
| 458 | 502 | }); |
| 459 | 503 | |
| 460 | - it("caches and restores via the runner cache endpoint, and round-trips artifacts", async () => { | |
| 504 | + it("caches and restores via the cache service, and round-trips artifacts", async () => { | |
| 461 | 505 | const steps: StepSpec[] = [ |
| 462 | 506 | { run: 'mkdir -p deps dist/sub && echo "${{ github.run_id }}" > deps/marker && echo a > dist/a.txt && echo b > dist/sub/b.txt' }, |
| 463 | 507 | { id: "cache", uses: "actions/cache@v4", with: { path: "deps", key: "deps-${{ hashFiles('**/lock.txt') }}-v1", "restore-keys": "deps-" } }, |
| @@ -470,11 +514,12 @@ | ||
| 470 | 514 | writeFileSync(path.join(workspace(), "lock.txt"), "v1"); |
| 471 | 515 | const c1 = await run(makeSpec(api, steps, { runId: "first" })); |
| 472 | 516 | expect(c1.conclusion, api.logText()).toBe("success"); |
| 473 | - expect(api.logText(2)).toContain("hit=false"); | |
| 517 | + expect(api.logText(2)).toMatch(/\nhit=\n/); // like actions/cache, a miss leaves cache-hit unset | |
| 474 | 518 | expect([...api.cache.keys()]).toHaveLength(1); |
| 475 | 519 | const key = [...api.cache.keys()][0]!; |
| 476 | 520 | expect(key).toMatch(/^deps-[0-9a-f]{64}-v1$/); |
| 477 | - expect(api.artifacts.has("dist")).toBe(true); | |
| 521 | + expect(api.artifacts.get("dist")?.mimeType).toBe("application/zip"); | |
| 522 | + expect(api.artifacts.get("dist")?.hash).toMatch(/^sha256:[0-9a-f]{64}$/); | |
| 478 | 523 | expect(api.logText(5)).toMatch(/a\nb\ndl=.*\/out/); |
| 479 | 524 | // post step saved the cache |
| 480 | 525 | expect(api.steps.some((s) => s.report.name === "Post Run actions/cache@v4" && s.report.conclusion === "success")).toBe(true); |
| @@ -488,7 +533,9 @@ | ||
| 488 | 533 | expect(c2.conclusion, api.logText()).toBe("success"); |
| 489 | 534 | expect(api.logText(1)).toContain("hit=true"); |
| 490 | 535 | expect(api.logText(1)).toContain("first"); |
| 491 | - expect(api.steps.some((s) => s.report.name?.startsWith("Post"))).toBe(false); | |
| 536 | + // like actions/cache, the post step runs but does not save over an exact hit | |
| 537 | + expect(api.logText()).toContain(`Cache hit occurred on the primary key ${key}, not saving cache.`); | |
| 538 | + expect([...api.cache.keys()]).toEqual([key]); | |
| 492 | 539 | }); |
| 493 | 540 | |
| 494 | 541 | it("checks out a repository with actions/checkout", async () => { |
| @@ -525,6 +572,145 @@ | ||
| 525 | 572 | rmSync(origin, { recursive: true, force: true }); |
| 526 | 573 | }); |
| 527 | 574 | |
| 575 | + it("checks out with LFS, submodules and sparse-checkout", async () => { | |
| 576 | + const root = tmp("checkout-features-"); | |
| 577 | + const gitIn = (cwd: string, ...args: string[]) => | |
| 578 | + execFileSync("git", ["-c", "protocol.file.allow=always", ...args], { cwd, env: { ...process.env, GIT_AUTHOR_NAME: "t", GIT_AUTHOR_EMAIL: "t@t", GIT_COMMITTER_NAME: "t", GIT_COMMITTER_EMAIL: "t@t" } }).toString().trim(); | |
| 579 | + const sub = path.join(root, "sub"); | |
| 580 | + const origin = path.join(root, "origin"); | |
| 581 | + mkdirSync(sub); | |
| 582 | + gitIn(sub, "init", "-q", "-b", "main"); | |
| 583 | + writeFileSync(path.join(sub, "lib.txt"), "from submodule\n"); | |
| 584 | + gitIn(sub, "add", "."); | |
| 585 | + gitIn(sub, "commit", "-q", "-m", "sub"); | |
| 586 | + mkdirSync(path.join(origin, "a"), { recursive: true }); | |
| 587 | + mkdirSync(path.join(origin, "b")); | |
| 588 | + gitIn(origin, "init", "-q", "-b", "main"); | |
| 589 | + gitIn(origin, "lfs", "install", "--local"); | |
| 590 | + gitIn(origin, "lfs", "track", "*.bin"); | |
| 591 | + writeFileSync(path.join(origin, "a/x.txt"), "x\n"); | |
| 592 | + writeFileSync(path.join(origin, "b/y.txt"), "y\n"); | |
| 593 | + writeFileSync(path.join(origin, "top.txt"), "top\n"); | |
| 594 | + writeFileSync(path.join(origin, "big.bin"), "real lfs content\n"); | |
| 595 | + gitIn(origin, "submodule", "add", "-q", `file://${sub}`, "vendor/sub"); | |
| 596 | + gitIn(origin, "add", "."); | |
| 597 | + gitIn(origin, "commit", "-q", "-m", "init"); | |
| 598 | + const sha = gitIn(origin, "rev-parse", "HEAD"); | |
| 599 | + const spec = makeSpec( | |
| 600 | + api, | |
| 601 | + [ | |
| 602 | + { uses: "actions/checkout@v4", with: { path: "full", lfs: "true", submodules: "recursive", "persist-credentials": "false" } }, | |
| 603 | + { run: "cat full/big.bin full/vendor/sub/lib.txt full/b/y.txt" }, | |
| 604 | + { uses: "actions/checkout@v4", with: { path: "plain" } }, | |
| 605 | + { run: "head -n 1 plain/big.bin; ls plain/vendor/sub | wc -l" }, | |
| 606 | + { uses: "actions/checkout@v4", with: { path: "sparse", "sparse-checkout": "a" } }, | |
| 607 | + { run: "ls sparse; ls sparse/a" }, | |
| 608 | + { uses: "actions/checkout@v4", with: { path: "nocone", "sparse-checkout": "b/y.txt", "sparse-checkout-cone-mode": "false" } }, | |
| 609 | + { run: "find nocone -type f -not -path '*/.git/*' | sort" }, | |
| 610 | + ], | |
| 611 | + { | |
| 612 | + sha, | |
| 613 | + repo: { owner: "acme", name: "widgets", cloneUrl: `file://${origin}`, defaultBranch: "main" }, | |
| 614 | + env: { GIT_CONFIG_COUNT: "1", GIT_CONFIG_KEY_0: "protocol.file.allow", GIT_CONFIG_VALUE_0: "always" }, | |
| 615 | + }, | |
| 616 | + ); | |
| 617 | + try { | |
| 618 | + const c = await run(spec); | |
| 619 | + expect(c.conclusion, api.logText()).toBe("success"); | |
| 620 | + expect(api.logText(1)).toContain("real lfs content\nfrom submodule\ny"); | |
| 621 | + expect(api.logText(3)).toContain("version https://git-lfs.github.com/spec/v1"); | |
| 622 | + expect(api.logText(3)).toMatch(/\n0$/); | |
| 623 | + expect(api.logText(5)).toMatch(/\na\nbig\.bin\ntop\.txt\nx\.txt$/); // cone mode keeps root files, drops b/ and vendor/ | |
| 624 | + expect(api.logText(7)).toMatch(/\nnocone\/b\/y\.txt$/); | |
| 625 | + } finally { | |
| 626 | + rmSync(root, { recursive: true, force: true }); | |
| 627 | + } | |
| 628 | + }); | |
| 629 | + | |
| 630 | + it("keeps the job token out of .git/config unless persist-credentials, even when checkout fails", async () => { | |
| 631 | + const root = tmp("http-git-"); | |
| 632 | + const origin = path.join(root, "acme/widgets.git"); | |
| 633 | + mkdirSync(origin, { recursive: true }); | |
| 634 | + const seed = path.join(root, "seed"); | |
| 635 | + const g = (cwd: string, ...args: string[]) => execFileSync("git", args, { cwd, env: { ...process.env, GIT_AUTHOR_NAME: "t", GIT_AUTHOR_EMAIL: "t@t", GIT_COMMITTER_NAME: "t", GIT_COMMITTER_EMAIL: "t@t" } }).toString().trim(); | |
| 636 | + g(origin, "init", "-q", "--bare", "-b", "main"); | |
| 637 | + mkdirSync(seed); | |
| 638 | + g(seed, "init", "-q", "-b", "main"); | |
| 639 | + writeFileSync(path.join(seed, "README.md"), "hello\n"); | |
| 640 | + g(seed, "add", "."); | |
| 641 | + g(seed, "commit", "-q", "-m", "first"); | |
| 642 | + g(seed, "push", "-q", origin, "main"); | |
| 643 | + const sha = g(seed, "rev-parse", "HEAD"); | |
| 644 | + // git's smart HTTP through `git http-backend`, recording the credentials it is sent | |
| 645 | + const auths: string[] = []; | |
| 646 | + const server = createServer((req, res) => { | |
| 647 | + auths.push(req.headers.authorization ?? ""); | |
| 648 | + const u = new URL(req.url ?? "/", "http://x"); | |
| 649 | + const cgi = spawn("git", ["http-backend"], { | |
| 650 | + env: { ...process.env, GIT_PROJECT_ROOT: root, GIT_HTTP_EXPORT_ALL: "1", PATH_INFO: u.pathname, QUERY_STRING: u.search.slice(1), REQUEST_METHOD: req.method ?? "GET", CONTENT_TYPE: req.headers["content-type"] ?? "", HTTP_GIT_PROTOCOL: String(req.headers["git-protocol"] ?? "") }, | |
| 651 | + }); | |
| 652 | + req.pipe(cgi.stdin); | |
| 653 | + let buf = Buffer.alloc(0); | |
| 654 | + cgi.stdout.on("data", (d: Buffer) => (buf = Buffer.concat([buf, d]))); | |
| 655 | + cgi.on("close", () => { | |
| 656 | + const sep = buf.indexOf("\r\n\r\n"); | |
| 657 | + const head = buf.subarray(0, sep).toString(); | |
| 658 | + const headers: Record<string, string> = {}; | |
| 659 | + let status = 200; | |
| 660 | + for (const line of head.split("\r\n")) { | |
| 661 | + const [k, ...v] = line.split(":"); | |
| 662 | + if (k!.toLowerCase() === "status") status = Number(v.join(":").trim().split(" ")[0]); | |
| 663 | + else headers[k!] = v.join(":").trim(); | |
| 664 | + } | |
| 665 | + res.writeHead(status, headers); | |
| 666 | + res.end(buf.subarray(sep + 4)); | |
| 667 | + }); | |
| 668 | + }); | |
| 669 | + await new Promise<void>((r) => server.listen(0, "127.0.0.1", () => r())); | |
| 670 | + const url = `http://127.0.0.1:${(server.address() as AddressInfo).port}`; | |
| 671 | + try { | |
| 672 | + const headers = "git config --get-regexp 'http\\..*\\.extraheader' | wc -l"; | |
| 673 | + const spec = makeSpec( | |
| 674 | + api, | |
| 675 | + [ | |
| 676 | + { uses: "actions/checkout@v4", with: { path: "a", "persist-credentials": "false" } }, | |
| 677 | + { run: `echo "a=$(cd a && ${headers})"; cat a/README.md` }, | |
| 678 | + { uses: "actions/checkout@v4", with: { path: "b" } }, | |
| 679 | + { run: `echo "b=$(cd b && ${headers})"` }, | |
| 680 | + { uses: "actions/checkout@v4", "continue-on-error": true, with: { path: "c", ref: "no-such-branch" } }, | |
| 681 | + { run: `echo "c=$(cd c && ${headers})"` }, | |
| 682 | + ], | |
| 683 | + { sha, repo: { owner: "acme", name: "widgets", cloneUrl: `${url}/acme/widgets.git`, defaultBranch: "main" } }, | |
| 684 | + ); | |
| 685 | + const c = await run(spec); | |
| 686 | + expect(c.conclusion, api.logText()).toBe("success"); | |
| 687 | + // the token reached the server while fetching... | |
| 688 | + expect(auths.filter((a) => a.toLowerCase().startsWith("basic ")).length).toBeGreaterThan(0); | |
| 689 | + expect(api.logText(1)).toContain("hello"); | |
| 690 | + // ...but stays in .git/config only when persisted, and not after a failed checkout | |
| 691 | + expect(api.logText(1)).toContain("a=0"); | |
| 692 | + expect(api.logText(3)).toContain("b=1"); | |
| 693 | + expect(api.logText(4)).toContain("Failed to fetch ref 'no-such-branch'"); | |
| 694 | + expect(api.logText(5)).toContain("c=0"); | |
| 695 | + } finally { | |
| 696 | + await new Promise<void>((r) => server.close(() => r())); | |
| 697 | + rmSync(root, { recursive: true, force: true }); | |
| 698 | + } | |
| 699 | + }); | |
| 700 | + | |
| 701 | + it("gives the runtime services' variables to actions, not to run steps, and never the runner's own token", async () => { | |
| 702 | + const c = await runAgentJob(makeSpec(api, [{ run: 'echo "rt=${ACTIONS_RUNTIME_TOKEN:-unset} results=${ACTIONS_RESULTS_URL:-unset} gh=${DIFFINERY_GITHUB_TOKEN:-unset} job=${DIFFINERY_JOB_TOKEN:-unset}"' }]), { | |
| 703 | + apiBase: apiBaseFor(api.url, "job-1"), | |
| 704 | + workRoot, | |
| 705 | + toolCache: path.join(workRoot, "_tool"), | |
| 706 | + logIntervalMs: 50, | |
| 707 | + echoLogs: false, | |
| 708 | + baseEnv: { ...process.env, DIFFINERY_GITHUB_TOKEN: "ghp_operator", DIFFINERY_JOB_TOKEN: "djt_x" }, | |
| 709 | + }); | |
| 710 | + expect(c.conclusion, api.logText()).toBe("success"); | |
| 711 | + expect(api.logText(0)).toContain("rt=unset results=unset gh=unset job=unset"); | |
| 712 | + }); | |
| 713 | + | |
| 528 | 714 | it("sets OIDC request variables when id-token is granted", async () => { |
| 529 | 715 | const c = await run( |
| 530 | 716 | makeSpec(api, [{ run: 'curl -s -H "Authorization: Bearer $ACTIONS_ID_TOKEN_REQUEST_TOKEN" "$ACTIONS_ID_TOKEN_REQUEST_URL&audience=sts" ; echo' }], { idToken: true }), |
addedcontainers/runner/test/runtime.test.ts+191−0View file
| @@ -1,0 +1,191 @@ | ||
| 1 | +import { execFileSync } from "node:child_process"; | |
| 2 | +import { chmodSync, mkdirSync, readFileSync, rmSync, statSync, writeFileSync } from "node:fs"; | |
| 3 | +import { createServer } from "node:http"; | |
| 4 | +import type { AddressInfo } from "node:net"; | |
| 5 | +import path from "node:path"; | |
| 6 | +import { afterEach, describe, expect, it } from "vitest"; | |
| 7 | +import { getBuiltin } from "../src/builtins/index.js"; | |
| 8 | +import { backendIds, findFilesToUpload, mimeTypeOf, nameMatches } from "../src/builtins/artifacts.js"; | |
| 9 | +import { cacheVersion, isExactKeyMatch } from "../src/builtins/cache.js"; | |
| 10 | +import { RuntimeClient, RuntimeError } from "../src/runtime.js"; | |
| 11 | +import { crc32, extractZip, safeEntryPath, writeZip } from "../src/zip.js"; | |
| 12 | +import { tmp } from "./helpers.js"; | |
| 13 | + | |
| 14 | +const dirs: string[] = []; | |
| 15 | +const scratch = (prefix: string) => { | |
| 16 | + const d = tmp(prefix); | |
| 17 | + dirs.push(d); | |
| 18 | + return d; | |
| 19 | +}; | |
| 20 | +afterEach(() => { | |
| 21 | + for (const d of dirs.splice(0)) rmSync(d, { recursive: true, force: true }); | |
| 22 | +}); | |
| 23 | + | |
| 24 | +const write = (root: string, rel: string, body: string | Buffer = rel) => { | |
| 25 | + mkdirSync(path.dirname(path.join(root, rel)), { recursive: true }); | |
| 26 | + writeFileSync(path.join(root, rel), body); | |
| 27 | +}; | |
| 28 | + | |
| 29 | +describe("zip", () => { | |
| 30 | + it("computes CRC-32", () => { | |
| 31 | + expect(crc32(Buffer.from("123456789")).toString(16)).toBe("cbf43926"); | |
| 32 | + expect(crc32(Buffer.from("6789"), crc32(Buffer.from("12345")))).toBe(crc32(Buffer.from("123456789"))); | |
| 33 | + }); | |
| 34 | + | |
| 35 | + it("writes archives unzip reads, and reads archives zip writes", async () => { | |
| 36 | + const d = scratch("zip-"); | |
| 37 | + write(d, "src/a.txt", "alpha\n".repeat(1000)); | |
| 38 | + write(d, "src/sub/b.bin", Buffer.alloc(70_000, 7)); | |
| 39 | + write(d, "src/empty", ""); | |
| 40 | + write(d, "src/run.sh", "#!/bin/sh\necho hi\n"); | |
| 41 | + chmodSync(path.join(d, "src/run.sh"), 0o755); | |
| 42 | + const entries = ["a.txt", "sub/b.bin", "empty", "run.sh"].map((name) => ({ source: path.join(d, "src", name), name })); | |
| 43 | + for (const level of [0, 6]) { | |
| 44 | + const out = path.join(d, `l${level}.zip`); | |
| 45 | + const { size, sha256 } = await writeZip(out, entries, level); | |
| 46 | + expect(size).toBe(statSync(out).size); | |
| 47 | + expect(sha256).toMatch(/^[0-9a-f]{64}$/); | |
| 48 | + expect(execFileSync("unzip", ["-t", out]).toString()).toContain("No errors detected"); | |
| 49 | + const dest = path.join(d, `x${level}`); | |
| 50 | + expect(await extractZip(out, dest)).toBe(4); | |
| 51 | + expect(readFileSync(path.join(dest, "a.txt"), "utf8")).toBe("alpha\n".repeat(1000)); | |
| 52 | + expect(readFileSync(path.join(dest, "sub/b.bin")).equals(Buffer.alloc(70_000, 7))).toBe(true); | |
| 53 | + expect(readFileSync(path.join(dest, "empty")).length).toBe(0); | |
| 54 | + expect(statSync(path.join(dest, "run.sh")).mode & 0o111).not.toBe(0); | |
| 55 | + } | |
| 56 | + execFileSync("zip", ["-qr", path.join(d, "cli.zip"), "."], { cwd: path.join(d, "src") }); | |
| 57 | + const dest = path.join(d, "cli"); | |
| 58 | + await extractZip(path.join(d, "cli.zip"), dest); | |
| 59 | + expect(readFileSync(path.join(dest, "sub/b.bin")).length).toBe(70_000); | |
| 60 | + }); | |
| 61 | + | |
| 62 | + it("refuses entries that escape the destination", () => { | |
| 63 | + expect(() => safeEntryPath("/tmp/x", "../evil")).toThrow(); | |
| 64 | + expect(() => safeEntryPath("/tmp/x", "/etc/passwd")).toThrow(); | |
| 65 | + expect(safeEntryPath("/tmp/x", "a/b.txt")).toBe("/tmp/x/a/b.txt"); | |
| 66 | + }); | |
| 67 | +}); | |
| 68 | + | |
| 69 | +describe("built-in actions", () => { | |
| 70 | + it("mirror only the refs and inputs they understand", () => { | |
| 71 | + expect(getBuiltin("actions/checkout", "v4")).not.toBeNull(); | |
| 72 | + expect(getBuiltin("Actions/Checkout", "v4.2.1", { "fetch-depth": "0", LFS: "true" })).not.toBeNull(); | |
| 73 | + expect(getBuiltin("actions/checkout", "main")).toBeNull(); | |
| 74 | + expect(getBuiltin("actions/checkout", "11bd71901bbe5b1630ceea73d27597364c9af683")).toBeNull(); | |
| 75 | + expect(getBuiltin("actions/checkout", "v99")).toBeNull(); | |
| 76 | + expect(getBuiltin("actions/checkout", "v4", { "ssh-key": "k" })).toBeNull(); | |
| 77 | + expect(getBuiltin("actions/cache", "v4", { path: "x", key: "k", "new-input": "1" })).toBeNull(); | |
| 78 | + expect(getBuiltin("actions/setup-node", "v4", { cache: "bun" })).toBeNull(); | |
| 79 | + expect(getBuiltin("actions/setup-python", "v5", { "python-version": "pypy3.10" })).toBeNull(); | |
| 80 | + expect(getBuiltin("actions/setup-python", "v5", { cache: "poetry" })).toBeNull(); | |
| 81 | + expect(getBuiltin("actions/github-script", "v7")).toBeNull(); | |
| 82 | + }); | |
| 83 | + | |
| 84 | + it("step aside when the workflow or runner asks for the real action", () => { | |
| 85 | + expect(getBuiltin("actions/cache", "v4", {}, { DIFFINERY_BUILTINS: "false" })).toBeNull(); | |
| 86 | + expect(getBuiltin("actions/cache", "v4", {}, { DIFFINERY_BUILTINS: "true" })).not.toBeNull(); | |
| 87 | + process.env.DIFFINERY_DISABLE_BUILTINS = "1"; | |
| 88 | + try { | |
| 89 | + expect(getBuiltin("actions/cache", "v4")).toBeNull(); | |
| 90 | + } finally { | |
| 91 | + delete process.env.DIFFINERY_DISABLE_BUILTINS; | |
| 92 | + } | |
| 93 | + }); | |
| 94 | +}); | |
| 95 | + | |
| 96 | +describe("cache", () => { | |
| 97 | + it("versions entries like @actions/cache", () => { | |
| 98 | + // sha256("node_modules|zstd-without-long|1.0"), computed independently | |
| 99 | + const expected = execFileSync("sh", ["-c", "printf '%s' 'node_modules|zstd-without-long|1.0' | sha256sum"]).toString().split(" ")[0]; | |
| 100 | + expect(cacheVersion(["node_modules"], "zstd-without-long")).toBe(expected); | |
| 101 | + expect(cacheVersion(["a", "b"], "gzip")).not.toBe(cacheVersion(["b", "a"], "gzip")); | |
| 102 | + // gzip is part of the version too: @actions/cache's getCacheVersion pushes the compression | |
| 103 | + // method whatever it is (only enableCrossOsArchive adds anything else) | |
| 104 | + const gzip = execFileSync("sh", ["-c", "printf '%s' 'node_modules|gzip|1.0' | sha256sum"]).toString().split(" ")[0]; | |
| 105 | + expect(cacheVersion(["node_modules"], "gzip")).toBe(gzip); | |
| 106 | + }); | |
| 107 | + | |
| 108 | + it("matches keys case-insensitively, like actions/cache", () => { | |
| 109 | + expect(isExactKeyMatch("Linux-deps", "linux-DEPS")).toBe(true); | |
| 110 | + expect(isExactKeyMatch("linux-deps", "linux-deps-2")).toBe(false); | |
| 111 | + expect(isExactKeyMatch("linux-deps", undefined)).toBe(false); | |
| 112 | + }); | |
| 113 | +}); | |
| 114 | + | |
| 115 | +describe("artifacts", () => { | |
| 116 | + it("finds files and their root like upload-artifact", () => { | |
| 117 | + const ws = scratch("upload-"); | |
| 118 | + for (const f of ["dist/a.js", "dist/sub/b.js", "dist/.env", "dist/.cache/x", "logs/one.log", "logs/two.log", "README.md"]) write(ws, f); | |
| 119 | + const rel = (r: { files: string[]; root: string }) => ({ files: r.files.map((f) => path.relative(ws, f)).sort(), root: path.relative(ws, r.root) }); | |
| 120 | + expect(rel(findFilesToUpload(ws, "dist/", false))).toEqual({ files: ["dist/a.js", "dist/sub/b.js"], root: "dist" }); | |
| 121 | + expect(rel(findFilesToUpload(ws, "dist", true)).files).toEqual(["dist/.cache/x", "dist/.env", "dist/a.js", "dist/sub/b.js"]); | |
| 122 | + expect(rel(findFilesToUpload(ws, "README.md", false))).toEqual({ files: ["README.md"], root: "" }); | |
| 123 | + expect(rel(findFilesToUpload(ws, "dist/sub/b.js\nlogs/*.log\n!logs/two.log", false))).toEqual({ files: ["dist/sub/b.js", "logs/one.log"], root: "" }); | |
| 124 | + expect(rel(findFilesToUpload(ws, "dist/**/*.js", false))).toEqual({ files: ["dist/a.js", "dist/sub/b.js"], root: "dist" }); | |
| 125 | + expect(findFilesToUpload(ws, "missing/", false).files).toEqual([]); | |
| 126 | + }); | |
| 127 | + | |
| 128 | + it("matches artifact names like minimatch", () => { | |
| 129 | + expect(nameMatches("build-*", "build-linux")).toBe(true); | |
| 130 | + expect(nameMatches("build-*", "test-linux")).toBe(false); | |
| 131 | + expect(nameMatches("{a,b}-?", "b-1")).toBe(true); | |
| 132 | + expect(nameMatches("cov[0-9]", "cov7")).toBe(true); | |
| 133 | + }); | |
| 134 | + | |
| 135 | + it("knows the run and job from the runtime token, and guesses mime types", () => { | |
| 136 | + const b64 = (v: unknown) => Buffer.from(JSON.stringify(v)).toString("base64url"); | |
| 137 | + expect(backendIds(`${b64({})}.${b64({ scp: "Actions.GenericRead:12 Actions.Results:12:job-9" })}.sig`)).toEqual({ run: "12", job: "job-9" }); | |
| 138 | + expect(() => backendIds(`${b64({})}.${b64({ scp: "Actions.GenericRead:12" })}.sig`)).toThrow(); | |
| 139 | + expect(mimeTypeOf("report.HTML")).toBe("text/html"); | |
| 140 | + expect(mimeTypeOf("blob.weird")).toBe("application/octet-stream"); | |
| 141 | + }); | |
| 142 | +}); | |
| 143 | + | |
| 144 | +describe("runtime client", () => { | |
| 145 | + it("uploads large files as blocks, small ones in one request, and downloads them back", async () => { | |
| 146 | + const store = new Map<string, Buffer>(); | |
| 147 | + const calls: string[] = []; | |
| 148 | + let blocks = new Map<string, Buffer>(); | |
| 149 | + const server = createServer(async (req, res) => { | |
| 150 | + const chunks: Buffer[] = []; | |
| 151 | + for await (const c of req) chunks.push(c as Buffer); | |
| 152 | + const body = Buffer.concat(chunks); | |
| 153 | + const u = new URL(req.url!, "http://x"); | |
| 154 | + const comp = u.searchParams.get("comp"); | |
| 155 | + calls.push(`${req.method} ${comp ?? "-"}`); | |
| 156 | + if (req.method === "PUT" && comp === "block") blocks.set(u.searchParams.get("blockid")!, body); | |
| 157 | + else if (req.method === "PUT" && comp === "blocklist") { | |
| 158 | + const ids = [...body.toString().matchAll(/<Latest>([^<]+)<\/Latest>/g)].map((m) => m[1]!); | |
| 159 | + store.set(u.pathname, Buffer.concat(ids.map((id) => blocks.get(id)!))); | |
| 160 | + blocks = new Map(); | |
| 161 | + } else if (req.method === "PUT") store.set(u.pathname, body); | |
| 162 | + else { | |
| 163 | + const b = store.get(u.pathname); | |
| 164 | + res.writeHead(b ? 200 : 404, { "content-type": "application/zip", "content-disposition": "attachment; filename*=UTF-8''r%C3%A9sum%C3%A9.zip" }); | |
| 165 | + return res.end(b); | |
| 166 | + } | |
| 167 | + res.writeHead(201); | |
| 168 | + res.end(); | |
| 169 | + }); | |
| 170 | + await new Promise<void>((r) => server.listen(0, "127.0.0.1", () => r())); | |
| 171 | + const base = `http://127.0.0.1:${(server.address() as AddressInfo).port}`; | |
| 172 | + const d = scratch("blocks-"); | |
| 173 | + try { | |
| 174 | + const rt = new RuntimeClient("t", `${base}/`, { retries: 0 }); | |
| 175 | + const big = Buffer.alloc(33 * 1024 * 1024 + 5); | |
| 176 | + for (let i = 0; i < big.length; i += 4096) big[i] = i % 251; | |
| 177 | + writeFileSync(path.join(d, "big"), big); | |
| 178 | + writeFileSync(path.join(d, "small"), "hello"); | |
| 179 | + expect(await rt.upload(`${base}/big?sig=1`, path.join(d, "big"))).toBe(big.length); | |
| 180 | + expect(calls).toEqual(["PUT block", "PUT block", "PUT blocklist"]); | |
| 181 | + expect(await rt.upload(`${base}/small?sig=1`, path.join(d, "small"))).toBe(5); | |
| 182 | + expect(calls.at(-1)).toBe("PUT -"); | |
| 183 | + const meta = await rt.download(`${base}/big?sig=1`, path.join(d, "back")); | |
| 184 | + expect(meta).toEqual({ contentType: "application/zip", fileName: "résumé.zip" }); | |
| 185 | + expect(readFileSync(path.join(d, "back")).equals(big)).toBe(true); | |
| 186 | + await expect(rt.download(`${base}/nothing`, path.join(d, "none"))).rejects.toBeInstanceOf(RuntimeError); | |
| 187 | + } finally { | |
| 188 | + await new Promise<void>((r) => server.close(() => r())); | |
| 189 | + } | |
| 190 | + }); | |
| 191 | +}); |
modifiedscripts/bootstrap.sh+22−4View file
| @@ -56,11 +56,29 @@ | ||
| 56 | 56 | cf POST /r2/buckets '{"name":"diffinery-data"}' | jq -e '.success' >/dev/null || { echo "could not create R2 bucket (R2 enabled? token has R2: Edit?)" >&2; exit 1; } |
| 57 | 57 | echo "created R2 bucket diffinery-data" |
| 58 | 58 | fi |
| 59 | -# Expire transient data: run logs/caches are rewritten on use, artifacts and event archives age out. | |
| 60 | -cf PUT "/r2/buckets/diffinery-data/lifecycle" '{"rules":[ | |
| 61 | - {"id":"artifacts-90d","enabled":true,"conditions":{"prefix":"repos/"},"deleteObjectsTransition":{"condition":{"type":"Age","maxAge":7776000}},"abortMultipartUploadsTransition":{"condition":{"type":"Age","maxAge":86400}}}, | |
| 59 | +# Lifecycle rules (layout: apps/diffinery/src/lib/storage-keys.ts). Only transient prefixes | |
| 60 | +# age out; repos/ (git objects, LFS, release assets), registry/ and actions-cache/ (evicted | |
| 61 | +# by Diffinery) have no age rule. Multipart uploads left unfinished are aborted after a day. | |
| 62 | +LIFECYCLE='{"rules":[ | |
| 63 | + {"id":"abort-multipart-1d","enabled":true,"conditions":{"prefix":""},"abortMultipartUploadsTransition":{"condition":{"type":"Age","maxAge":86400}}}, | |
| 64 | + {"id":"artifacts-90d","enabled":true,"conditions":{"prefix":"artifacts/"},"deleteObjectsTransition":{"condition":{"type":"Age","maxAge":7776000}}}, | |
| 65 | + {"id":"logs-90d","enabled":true,"conditions":{"prefix":"logs/"},"deleteObjectsTransition":{"condition":{"type":"Age","maxAge":7776000}}}, | |
| 66 | + {"id":"actions-staging-1d","enabled":true,"conditions":{"prefix":"actions-staging/"},"deleteObjectsTransition":{"condition":{"type":"Age","maxAge":86400}}}, | |
| 67 | + {"id":"archives-30d","enabled":true,"conditions":{"prefix":"archives/"},"deleteObjectsTransition":{"condition":{"type":"Age","maxAge":2592000}}}, | |
| 62 | 68 | {"id":"events-400d","enabled":true,"conditions":{"prefix":"events/"},"deleteObjectsTransition":{"condition":{"type":"Age","maxAge":34560000}}} |
| 63 | -]}' >/dev/null || true | |
| 69 | +]}' | |
| 70 | +# Applied on every deploy; a failure stops it, since a wrong rule can delete data. | |
| 71 | +RESULT="$(cf PUT "/r2/buckets/diffinery-data/lifecycle" "$LIFECYCLE")" | |
| 72 | +if ! jq -e '.success' >/dev/null <<<"$RESULT"; then | |
| 73 | + echo "could not set the R2 lifecycle rules (token has R2: Edit?): $(jq -c '.errors // .' <<<"$RESULT" 2>/dev/null || echo "$RESULT")" >&2 | |
| 74 | + exit 1 | |
| 75 | +fi | |
| 76 | +CURRENT="$(cf GET "/r2/buckets/diffinery-data/lifecycle")" | |
| 77 | +if jq -e '[.result.rules[]? | select(.deleteObjectsTransition != null and ((.conditions.prefix // "") | startswith("repos/") or . == ""))] | length > 0' >/dev/null <<<"$CURRENT"; then | |
| 78 | + echo "an R2 lifecycle rule still deletes objects under repos/ (git objects, LFS); refusing to continue" >&2 | |
| 79 | + exit 1 | |
| 80 | +fi | |
| 81 | +echo "R2 lifecycle rules set" | |
| 64 | 82 | endlog |
| 65 | 83 | |
| 66 | 84 | # ------------------------------------------------------------------ Queues |