Diffinery

codician/diffinerypublic

Open-source, agent-first GitHub and GitHub Actions alternative that runs on your own Cloudflare account: git hosting, pull requests, code review and drop-in GitHub Actions YAML on Workers, Durable Objects, R2 and Containers.

Imports: split a refused batch of files again, and allow settings while an import runs (#181)

* Imports: a refused batch of files is split again, and settings can change while an import runs
test-bot authored 1 parent a233550commit 5594b9dd61969a269ce5ebbb16582c11f190070bBrowse files
10 files changed+105−49
Files (10)
modifiedapps/diffinery/docs/migrate-from-github.md+2−2View file
@@ -28,7 +28,7 @@
2828 Things to know:
2929
3030 - **Nothing the import writes fires events.** No workflow runs start, no webhooks are delivered and nobody is notified about imported issues, comments or pushes. Your first push after the import starts workflows as usual.
31-- **The repository is locked while the import runs.** Until the import finishes, fails or is cancelled, only the import writes to it. A push from anyone else is refused with `repository is locked by an import in progress`, and any other change through the REST API, the web UI or MCP (an issue, a comment, a label, a setting, a transfer) gets `423 repository is locked by an import in progress; try again when it finishes, or cancel the import`. Reading the repository and watching it work as usual.
31+- **The repository is locked while the import runs.** Until the import finishes, fails or is cancelled, only the import writes to it. A push from anyone else is refused with `repository is locked by an import in progress`, and any other change through the REST API, the web UI or MCP (an issue, a comment, a label, a webhook, a branch protection rule, a rename or transfer) gets `423 repository is locked by an import in progress; try again when it finishes, or cancel the import`. Reading the repository and watching it work as usual, and so do settings that don't touch its Git data, issues or pull requests: Actions secrets and variables, environments with their secrets and variables, and collaborators and their invitations. You can set the imported secrets' values while the history is still arriving.
3232 - **Every write is keyed by GitHub's ids**, so a retried step or a retried import never duplicates issues, pull requests, comments, reviews, labels, milestones, branch protection rules or releases (a release is matched by its tag and creation date, and assets already copied are kept).
3333 - **The import keeps running if you close the page or stop the CLI.** You get an entry in your notifications list when it succeeds (`Imported acme/website into acme/website`) or fails, linking to its page.
3434 - **Every import is audited:** `import.create`, `import.push`, `import.finish`, `import.fail`, `import.cancel` and `import.retry` appear in the audit log. See [Audit log](/docs/security/audit-log).
@@ -591,7 +591,7 @@
591591 - Finished work isn't repeated or duplicated: steps write by GitHub id, and the Git transfer only sends what the repository doesn't have yet.
592592 - An agent can't retry once the repository was created (`an agent cannot retry an import after the repository was created; a person who administers acme must retry it`). A person who administers the target retries it with **Retry this import** or the API, giving the GitHub token again if the source is private.
593593
594-**The repository lock.** From the moment the import creates the repository until it finishes, fails or is cancelled, only the import may write to it. Any other push gets `repository is locked by an import in progress`, and any other write through the REST API, the web UI or MCP gets `423 repository is locked by an import in progress; try again when it finishes, or cancel the import`. Watching the repository still works. The import itself may only create refs or move refs it wrote. A retry claims the repository again only when it holds nothing the import didn't write. If anyone pushed a branch, opened an issue, added a label, a protection rule or a release in the meantime (for example after the import failed and released the lock), the retry fails with `repository already holds content not written by this import; start a new import`. A failed or cancelled import leaves the partly imported repository in place; delete it before importing again under the same name.
594+**The repository lock.** From the moment the import creates the repository until it finishes, fails or is cancelled, only the import may write to it. Any other push gets `repository is locked by an import in progress`, and any other write through the REST API, the web UI or MCP gets `423 repository is locked by an import in progress; try again when it finishes, or cancel the import`. Watching the repository still works, and so do Actions secrets and variables, environments (with their secrets and variables), and collaborators and invitations. The import never writes secret values; its variables step writes GitHub's values, so a variable you set before that step under the same name ends with GitHub's value. The import itself may only create refs or move refs it wrote. A retry claims the repository again only when it holds nothing the import didn't write. If anyone pushed a branch, opened an issue, added a label, a protection rule or a release in the meantime (for example after the import failed and released the lock), the retry fails with `repository already holds content not written by this import; start a new import`. A failed or cancelled import leaves the partly imported repository in place; delete it before importing again under the same name.
595595
596596 ## Agents and imports
597597
modifiedapps/diffinery/src/http/api.ts+8−8View file
@@ -18,7 +18,7 @@
1818 import { imports } from "./imports.js";
1919 import { schedulerStub } from "../lib/schedules.js";
2020 import { agenticApi } from "../agentic/api.js";
21-import { auditRepo, hiddenAuthors, loadRepo, refuseAgent, repoStub } from "./common.js";
21+import { auditRepo, hiddenAuthors, loadRepo, refuseAgent, repoStub, WHILE_IMPORTING } from "./common.js";
2222 import { repoCreated } from "../mail/notify.js";
2323 import { actionsRest } from "./actions-runtime.js";
2424 import { compat } from "./compat.js";
@@ -349,7 +349,7 @@
349349 return c.json((await c.get("dir").invites({ repoId: repo.id, pending: true })).map(inviteJson));
350350 });
351351 api.delete("/repos/:owner/:repo/invitations/:id", async (c) => {
352- const { repo, caller } = await loadRepo(c, P(c).owner, P(c).repo, "admin");
352+ const { repo, caller } = await loadRepo(c, P(c).owner, P(c).repo, "admin", WHILE_IMPORTING);
353353 if (!(await c.get("dir").revokeInvite(c.req.param("id"), { repoId: repo.id }))) throw new HttpError(404, "no pending invitation with this id");
354354 auditRepo(c, caller, repo, "repo.invite.revoke", { id: c.req.param("id") });
355355 return c.body(null, 204);
@@ -456,7 +456,7 @@
456456 return c.json(paginate(c, (await c.get("dir").collaborators(repo.id)).map((x) => ({ ...userJson(c, x.account), permissions: permissionsOf(x.role), role_name: x.role }))));
457457 });
458458 api.put("/repos/:owner/:repo/collaborators/:login", async (c) => {
459- const { repo, caller } = await loadRepo(c, P(c).owner, P(c).repo, "admin");
459+ const { repo, caller } = await loadRepo(c, P(c).owner, P(c).repo, "admin", WHILE_IMPORTING);
460460 const who = await c.get("dir").accountByLogin(c.req.param("login"));
461461 if (!who) throw new HttpError(404, "user not found");
462462 if (who.kind === "org") throw new HttpError(422, "an organization can't be a collaborator: add its members, or transfer the repository to it");
@@ -553,7 +553,7 @@
553553 return c.json(membershipJson(c, me, pending.org!, pending.role, "active"));
554554 });
555555 api.delete("/repos/:owner/:repo/collaborators/:login", async (c) => {
556- const { repo, caller } = await loadRepo(c, P(c).owner, P(c).repo, "admin");
556+ const { repo, caller } = await loadRepo(c, P(c).owner, P(c).repo, "admin", WHILE_IMPORTING);
557557 const who = await c.get("dir").accountByLogin(c.req.param("login"));
558558 if (who) {
559559 await c.get("dir").setCollaborator(repo.id, who.id, null);
@@ -1140,7 +1140,7 @@
11401140 return c.json({ total_count: s.length, secrets: s.map((x) => ({ name: x.name, updated_at: x.updatedAt })) });
11411141 });
11421142 api.put("/repos/:owner/:repo/actions/secrets/:name", async (c) => {
1143- const { stub, caller, repo } = await loadRepo(c, P(c).owner, P(c).repo, "admin");
1143+ const { stub, caller, repo } = await loadRepo(c, P(c).owner, P(c).repo, "admin", WHILE_IMPORTING);
11441144 if (caller.kind === "agent") throw new HttpError(403, "agents cannot manage secrets");
11451145 const b = await body<{ value: string; encrypted_value?: string }>(c);
11461146 if (typeof b.value !== "string" && typeof b.encrypted_value === "string") throw new HttpError(422, "encrypted_value (sealed box) is not supported yet: send the secret as `value` over HTTPS; it is encrypted at rest");
@@ -1150,7 +1150,7 @@
11501150 return c.body(null, 204);
11511151 });
11521152 api.delete("/repos/:owner/:repo/actions/secrets/:name", async (c) => {
1153- const { stub, caller, repo } = await loadRepo(c, P(c).owner, P(c).repo, "admin");
1153+ const { stub, caller, repo } = await loadRepo(c, P(c).owner, P(c).repo, "admin", WHILE_IMPORTING);
11541154 refuseAgent(caller, "secrets");
11551155 await stub.deleteSecret(c.req.param("name"));
11561156 auditRepo(c, caller, repo, "repo.secret.delete", { name: c.req.param("name").toUpperCase() });
@@ -1163,14 +1163,14 @@
11631163 return c.json({ total_count: all.length, variables: paginate(c, all) });
11641164 });
11651165 api.delete("/repos/:owner/:repo/actions/variables/:name", async (c) => {
1166- const { stub, caller, repo } = await loadRepo(c, P(c).owner, P(c).repo, "admin");
1166+ const { stub, caller, repo } = await loadRepo(c, P(c).owner, P(c).repo, "admin", WHILE_IMPORTING);
11671167 refuseAgent(caller, "variables");
11681168 await stub.setVariable(c.req.param("name"), null);
11691169 auditRepo(c, caller, repo, "repo.variable.delete", { name: c.req.param("name").toUpperCase() });
11701170 return c.body(null, 204);
11711171 });
11721172 api.put("/repos/:owner/:repo/actions/variables/:name", async (c) => {
1173- const { stub, caller, repo } = await loadRepo(c, P(c).owner, P(c).repo, "admin");
1173+ const { stub, caller, repo } = await loadRepo(c, P(c).owner, P(c).repo, "admin", WHILE_IMPORTING);
11741174 refuseAgent(caller, "variables");
11751175 await stub.setVariable(c.req.param("name"), (await body<{ value: string }>(c)).value);
11761176 auditRepo(c, caller, repo, "repo.variable.set", { name: c.req.param("name").toUpperCase() });
modifiedapps/diffinery/src/http/common.ts+13−2View file
@@ -22,6 +22,16 @@
2222 }
2323
2424 /**
25+ * For `loadRepo`: a settings change that doesn't touch the repository's Git data, issues or pull
26+ * requests (secrets, variables, environments, collaborators and invitations) goes through while
27+ * an import fills the repository, so people can set it up as it arrives. An import never writes
28+ * secret values; its variables and environments step writes GitHub's, so one set here before that
29+ * step under the same name ends with GitHub's value. Webhooks stay locked: nothing the import
30+ * does may reach one.
31+ */
32+export const WHILE_IMPORTING = { whileImporting: true } as const;
33+
34+/**
2535 * Load a repository and the caller's role, enforcing a minimum role (404 hides
2636 * private repos; signed-out callers get the same answer for private and missing ones). Anonymous callers get read access to public repositories but
2737 * may only read: any other method needs a signed-in caller, as on GitHub,
@@ -31,8 +41,9 @@
3141 * except the few routes that pass `whileArchived` (unarchiving, deleting,
3242 * watching), as on GitHub. While a GitHub import fills the repository, every
3343 * write is refused with 423 except the import's own (its push token) and the
34- * routes that pass `whileImporting` (watching, and pushes, which the
35- * repository refuses itself with a message the git client shows).
44+ * routes that pass `whileImporting` (watching, settings that don't touch Git
45+ * data (see WHILE_IMPORTING), and pushes, which the repository refuses itself
46+ * with a message the git client shows).
3647 *
3748 * An owner/name that a repository was transferred or renamed away from leads
3849 * to it, after the same visibility check, so callers who can't read it still
modifiedapps/diffinery/src/http/deployments.ts+11−11View file
@@ -7,14 +7,14 @@
77 import type { Caller } from "../repo/types.js";
88 import { baseOf, ghUser, isoSeconds, nodeId, pageOf, paginate, repoApi, repoHtml, setLinks } from "./gh.js";
99 import { intParam, jsonBody, loadTarget, repoParams } from "./ghctx.js";
10-import { refuseAgent } from "./common.js";
10+import { refuseAgent, WHILE_IMPORTING } from "./common.js";
1111 import { resolveReviewers, reviewDeployments, viewerFor } from "./environment-reviews.js";
1212
1313 /** Deployments, deployment statuses and environments, with environment secrets and variables (GitHub REST). */
1414 export const deployments = new Hono<{ Bindings: Env; Variables: AppVars }>();
1515
1616 const R = "/repos/:owner/:repo";
17-const load = (c: AppContext, min: "read" | "write" | "admin" = "read") => loadTarget(c, min);
17+const load = (c: AppContext, min: "read" | "write" | "admin" = "read", opts: { whileImporting?: boolean } = {}) => loadTarget(c, min, opts);
1818 const api = (c: AppContext) => repoApi(baseOf(c), repoParams(c).owner, repoParams(c).repo);
1919 const envName = (c: AppContext) => (c.req.param("env") ?? "");
2020
@@ -174,7 +174,7 @@
174174 return c.json(environmentJson(c, await stub.environment(envName(c))));
175175 });
176176 deployments.put(`${R}/environments/:env`, async (c) => {
177- const { stub, caller, repo } = await load(c, "admin");
177+ const { stub, caller, repo } = await load(c, "admin", WHILE_IMPORTING);
178178 const b = await jsonBody<{ wait_timer?: unknown; reviewers?: unknown; deployment_branch_policy?: { protected_branches?: boolean; custom_branch_policies?: boolean } | null; prevent_self_review?: unknown }>(c, true);
179179 if (caller.kind === "agent") throw new HttpError(403, "only repository admins (people, not agents) can manage environments");
180180 const reviewers = await resolveReviewers(c, repo, b.reviewers);
@@ -189,7 +189,7 @@
189189 return c.json(environmentJson(c, e));
190190 });
191191 deployments.delete(`${R}/environments/:env`, async (c) => {
192- const { stub, caller } = await load(c, "admin");
192+ const { stub, caller } = await load(c, "admin", WHILE_IMPORTING);
193193 await stub.deleteEnvironment(envName(c), caller);
194194 audit(c, caller, "environment.delete", { environment: envName(c) });
195195 return c.body(null, 204);
@@ -201,14 +201,14 @@
201201 return c.json({ total_count: list.length, branch_policies: paginate(c, list).map((p) => ({ ...p, node_id: nodeId("BranchPolicy", `${e.id}/${p.id}`) })) });
202202 });
203203 deployments.post(`${R}/environments/:env/deployment-branch-policies`, async (c) => {
204- const { stub, caller } = await load(c, "admin");
204+ const { stub, caller } = await load(c, "admin", WHILE_IMPORTING);
205205 const b = await jsonBody<{ name?: string; type?: string }>(c);
206206 const p = await stub.addEnvironmentBranchPattern(envName(c), b, caller);
207207 audit(c, caller, "environment.branch_policy.create", { environment: envName(c), pattern: p.name, type: p.type });
208208 return c.json({ ...p, node_id: nodeId("BranchPolicy", p.id) });
209209 });
210210 deployments.delete(`${R}/environments/:env/deployment-branch-policies/:id`, async (c) => {
211- const { stub, caller } = await load(c, "admin");
211+ const { stub, caller } = await load(c, "admin", WHILE_IMPORTING);
212212 await stub.removeEnvironmentBranchPattern(envName(c), intParam(c, "id"), caller);
213213 audit(c, caller, "environment.branch_policy.delete", { environment: envName(c), id: intParam(c, "id") });
214214 return c.body(null, 204);
@@ -228,7 +228,7 @@
228228 return c.json({ name: s.name, created_at: isoSeconds(s.updatedAt), updated_at: isoSeconds(s.updatedAt) });
229229 });
230230 deployments.put(`${R}/environments/:env/secrets/:name`, async (c) => {
231- const { stub, caller } = await load(c, "admin");
231+ const { stub, caller } = await load(c, "admin", WHILE_IMPORTING);
232232 if (caller.kind === "agent") throw new HttpError(403, "agents cannot manage secrets");
233233 const b = await jsonBody<{ value?: string; encrypted_value?: string }>(c);
234234 if (typeof b.value !== "string" && typeof b.encrypted_value === "string") throw new HttpError(422, "encrypted_value (sealed box) is not supported yet: send the secret as `value` over HTTPS; it is encrypted at rest");
@@ -239,7 +239,7 @@
239239 return c.body(null, existed ? 204 : 201);
240240 });
241241 deployments.delete(`${R}/environments/:env/secrets/:name`, async (c) => {
242- const { stub, caller } = await load(c, "admin");
242+ const { stub, caller } = await load(c, "admin", WHILE_IMPORTING);
243243 if (caller.kind === "agent") throw new HttpError(403, "agents cannot manage secrets");
244244 await stub.deleteEnvironmentSecret(envName(c), c.req.param("name"), caller);
245245 audit(c, caller, "environment.secret.delete", { environment: envName(c), name: c.req.param("name").toUpperCase() });
@@ -260,7 +260,7 @@
260260 return c.json(variableJson(v));
261261 });
262262 deployments.post(`${R}/environments/:env/variables`, async (c) => {
263- const { stub, caller } = await load(c, "admin");
263+ const { stub, caller } = await load(c, "admin", WHILE_IMPORTING);
264264 refuseAgent(caller, "variables");
265265 const b = await jsonBody<{ name?: string; value?: string }>(c);
266266 if (typeof b.name !== "string" || typeof b.value !== "string") throw new HttpError(422, "name and value are required");
@@ -270,7 +270,7 @@
270270 return c.json({}, 201);
271271 });
272272 deployments.patch(`${R}/environments/:env/variables/:name`, async (c) => {
273- const { stub, caller } = await load(c, "admin");
273+ const { stub, caller } = await load(c, "admin", WHILE_IMPORTING);
274274 refuseAgent(caller, "variables");
275275 const b = await jsonBody<{ name?: string; value?: string }>(c);
276276 const name = c.req.param("name");
@@ -285,7 +285,7 @@
285285 return c.body(null, 204);
286286 });
287287 deployments.delete(`${R}/environments/:env/variables/:name`, async (c) => {
288- const { stub, caller } = await load(c, "admin");
288+ const { stub, caller } = await load(c, "admin", WHILE_IMPORTING);
289289 refuseAgent(caller, "variables");
290290 await stub.setEnvironmentVariable(envName(c), c.req.param("name"), null, caller);
291291 audit(c, caller, "environment.variable.delete", { environment: envName(c), name: c.req.param("name").toUpperCase() });
modifiedapps/diffinery/src/http/ghctx.ts+2−2View file
@@ -18,8 +18,8 @@
1818 * read-only for everyone (comments, reactions and settings included), so any
1919 * request that isn't a read is refused, whatever role the route needs.
2020 */
21-export async function loadTarget(c: AppContext, min: Role = "read"): Promise<Awaited<ReturnType<typeof loadRepo>>> {
22- const l = await loadRepo(c, c.req.param("owner") ?? "", c.req.param("repo") ?? "", min);
21+export async function loadTarget(c: AppContext, min: Role = "read", opts: { whileImporting?: boolean } = {}): Promise<Awaited<ReturnType<typeof loadRepo>>> {
22+ const l = await loadRepo(c, c.req.param("owner") ?? "", c.req.param("repo") ?? "", min, opts);
2323 if (l.repo.archived && !["GET", "HEAD", "OPTIONS"].includes(c.req.method)) throw new HttpError(403, "repository is archived (read-only)");
2424 return l;
2525 }
modifiedapps/diffinery/src/http/misc.ts+4−4View file
@@ -4,7 +4,7 @@
44 import { HttpError } from "../lib/util.js";
55 import { baseOf, ghRun, isoSeconds, nodeId, repoApi, repoHtml } from "./gh.js";
66 import { intParam, jsonBody, loadTarget, repoParams } from "./ghctx.js";
7-import { auditRepo, refuseAgent } from "./common.js";
7+import { auditRepo, refuseAgent, WHILE_IMPORTING } from "./common.js";
88
99 /**
1010 * Smaller GitHub endpoints scripts and actions call: rate limits, topics,
@@ -14,7 +14,7 @@
1414 export const misc = new Hono<{ Bindings: Env; Variables: AppVars }>();
1515
1616 const R = "/repos/:owner/:repo";
17-const load = (c: AppContext, min: "read" | "write" | "admin" = "read") => loadTarget(c, min);
17+const load = (c: AppContext, min: "read" | "write" | "admin" = "read", opts: { whileImporting?: boolean } = {}) => loadTarget(c, min, opts);
1818
1919 // Diffinery doesn't meter API calls; report a generous, constant budget so clients that check it carry on.
2020 const LIMIT = 15_000;
@@ -128,7 +128,7 @@
128128 return c.json({ name, value: vars[name], created_at: null, updated_at: null });
129129 });
130130 misc.post(`${R}/actions/variables`, async (c) => {
131- const { stub, caller, repo } = await load(c, "admin");
131+ const { stub, caller, repo } = await load(c, "admin", WHILE_IMPORTING);
132132 refuseAgent(caller, "variables");
133133 const b = await jsonBody<{ name?: string; value?: string }>(c);
134134 if (typeof b.name !== "string" || typeof b.value !== "string") throw new HttpError(422, "name and value are required", { errors: [{ resource: "ActionsVariable", code: "missing_field", field: typeof b.name !== "string" ? "name" : "value" }] });
@@ -138,7 +138,7 @@
138138 return c.json({}, 201);
139139 });
140140 misc.patch(`${R}/actions/variables/:name`, async (c) => {
141- const { stub, caller, repo } = await load(c, "admin");
141+ const { stub, caller, repo } = await load(c, "admin", WHILE_IMPORTING);
142142 refuseAgent(caller, "variables");
143143 const name = c.req.param("name");
144144 const vars = await stub.variables(false);
modifiedapps/diffinery/src/web/repo-settings.tsx+12−12View file
@@ -3,7 +3,7 @@
33 import { raw } from "hono/html";
44 import type { Child, FC } from "hono/jsx";
55 import { requireAccount, requireRecentSignIn, type AppContext } from "../auth/principal.js";
6-import { loadRepo } from "../http/common.js";
6+import { loadRepo, WHILE_IMPORTING } from "../http/common.js";
77 import { HttpError } from "../lib/util.js";
88 import { MAX_REVIEWERS, MAX_WAIT_TIMER, type Environment } from "../repo/deployments.js";
99 import { resolveReviewers } from "../http/environment-reviews.js";
@@ -20,7 +20,7 @@
2020
2121 const WEBHOOK_EVENTS = ["push", "pull_request", "pull_request_review", "issues", "issue_comment", "workflow_run", "workflow_dispatch"];
2222
23-const load = (c: AppContext, opts: { whileArchived?: boolean } = {}) => loadRepo(c, prm(c, "owner"), prm(c, "repo"), "admin", opts);
23+const load = (c: AppContext, opts: { whileArchived?: boolean; whileImporting?: boolean } = {}) => loadRepo(c, prm(c, "owner"), prm(c, "repo"), "admin", opts);
2424
2525 function audit(c: AppContext, l: Loaded, action: string, detail?: Record<string, unknown>): void {
2626 const p = c.get("principal");
@@ -818,7 +818,7 @@
818818 web.get("/:owner/:repo/settings/environments/:env", async (c) => environmentPage(c, await load(c), prm(c, "env")));
819819
820820 post(web, "/:owner/:repo/settings/environments", async (c, f) => {
821- const l = await load(c);
821+ const l = await load(c, WHILE_IMPORTING);
822822 const name = f.s("name").trim();
823823 return act(c, envPath(l), async () => {
824824 const e = await l.stub.upsertEnvironment(name, {}, l.caller);
@@ -828,7 +828,7 @@
828828 });
829829
830830 post(web, "/:owner/:repo/settings/environments/:env", async (c, f) => {
831- const l = await load(c);
831+ const l = await load(c, WHILE_IMPORTING);
832832 const name = prm(c, "env");
833833 return act(
834834 c,
@@ -850,7 +850,7 @@
850850 });
851851
852852 post(web, "/:owner/:repo/settings/environments/:env/delete", async (c) => {
853- const l = await load(c);
853+ const l = await load(c, WHILE_IMPORTING);
854854 const name = prm(c, "env");
855855 return act(
856856 c,
@@ -967,7 +967,7 @@
967967 });
968968
969969 post(web, "/:owner/:repo/settings/access", async (c, f) => {
970- const l = await load(c);
970+ const l = await load(c, WHILE_IMPORTING);
971971 const dir = c.get("dir");
972972 return act(c, `${repoUrl(l.repo)}/settings/access`, async () => {
973973 const who = await dir.accountByLogin(f.s("login").trim());
@@ -990,7 +990,7 @@
990990 });
991991
992992 post(web, "/:owner/:repo/settings/access/invites/:id/revoke", async (c) => {
993- const l = await load(c);
993+ const l = await load(c, WHILE_IMPORTING);
994994 return act(c, `${repoUrl(l.repo)}/settings/access`, async () => {
995995 if (!(await c.get("dir").revokeInvite(prm(c, "id"), { repoId: l.repo.id }))) throw new HttpError(404, "this invitation was already used, revoked or never existed");
996996 audit(c, l, "repo.invite.revoke", { id: prm(c, "id") });
@@ -998,7 +998,7 @@
998998 });
999999
10001000 post(web, "/:owner/:repo/settings/access/people", async (c, f) => {
1001- const l = await load(c);
1001+ const l = await load(c, WHILE_IMPORTING);
10021002 const me = requireAccount(c);
10031003 const role = ROLES.find((r) => r === f.s("role")) ?? "write";
10041004 let invited: { email: string; url: string } | undefined;
@@ -1051,7 +1051,7 @@
10511051 });
10521052
10531053 post(web, "/:owner/:repo/settings/secrets", async (c, f) => {
1054- const l = await load(c);
1054+ const l = await load(c, WHILE_IMPORTING);
10551055 return act(
10561056 c,
10571057 `${repoUrl(l.repo)}/settings/secrets`,
@@ -1071,7 +1071,7 @@
10711071 });
10721072
10731073 post(web, "/:owner/:repo/settings/secrets/delete", async (c, f) => {
1074- const l = await load(c);
1074+ const l = await load(c, WHILE_IMPORTING);
10751075 return act(c, `${repoUrl(l.repo)}/settings/secrets`, async () => {
10761076 await l.stub.deleteSecret(f.s("name"));
10771077 audit(c, l, "repo.secret.delete", { name: f.s("name") });
@@ -1079,7 +1079,7 @@
10791079 });
10801080
10811081 post(web, "/:owner/:repo/settings/variables", async (c, f) => {
1082- const l = await load(c);
1082+ const l = await load(c, WHILE_IMPORTING);
10831083 return act(c, `${repoUrl(l.repo)}/settings/secrets`, async () => {
10841084 const name = f.s("name").trim();
10851085 if (!/^[A-Za-z_][A-Za-z0-9_]*$/.test(name)) throw new HttpError(422, "invalid variable name");
@@ -1089,7 +1089,7 @@
10891089 });
10901090
10911091 post(web, "/:owner/:repo/settings/variables/delete", async (c, f) => {
1092- const l = await load(c);
1092+ const l = await load(c, WHILE_IMPORTING);
10931093 return act(c, `${repoUrl(l.repo)}/settings/secrets`, async () => {
10941094 await l.stub.setVariable(f.s("name"), null);
10951095 audit(c, l, "repo.variable.delete", { name: f.s("name") });
modifiedapps/diffinery/test/e2e/e2e.test.mjs+7−2View file
@@ -2498,8 +2498,9 @@
24982498 const started = await ok("POST", "/diffinery/imports", { source: "octo/app", owner, name: "imported-app", github_token: GH_TOKEN, github_api_url: apiUrl });
24992499 assert.equal(started.status, "queued");
25002500 assert.ok(!JSON.stringify(started).includes(GH_TOKEN), "the token is never returned");
2501- // While the import holds the repository it created, nothing else writes to it: the REST API, the
2502- // web UI and MCP get 423 Locked (reads still work). The import's own writes go through below.
2501+ // While the import holds the repository it created, nothing else writes to it but settings such as
2502+ // secrets: the REST API, the web UI and MCP get 423 Locked (reads still work). The import's own
2503+ // writes go through below.
25032504 await until(`import ${started.id} to wait for an agent`, async () => (await ok("GET", `/diffinery/imports/${started.id}`)).git?.phase === "waiting", 120_000);
25042505 const L = `/repos/${owner}/imported-app`;
25052506 for (const [method, path, body] of [
@@ -2514,6 +2515,10 @@
25142515 assert.match(r.json.message, /locked by an import/);
25152516 }
25162517 assert.equal((await api("GET", `${L}/issues`)).status, 200, "reads are not locked");
2518+ // Settings that don't touch Git data, issues or pull requests can be set up while it imports.
2519+ assert.equal((await api("PUT", `${L}/actions/secrets/EARLY_SECRET`, { value: "set during the import" })).status, 204);
2520+ assert.ok((await ok("GET", `${L}/actions/secrets`)).secrets.some((x) => x.name === "EARLY_SECRET"));
2521+ assert.equal((await api("DELETE", `${L}/actions/secrets/EARLY_SECRET`)).status, 204);
25172522 const csrf = randomBytes(16).toString("hex");
25182523 const webWrite = await fetch(`${base}/${owner}/imported-app/issues`, { method: "POST", headers: { cookie: `dfy_csrf=${csrf}`, "content-type": "application/x-www-form-urlencoded" }, body: new URLSearchParams({ _csrf: csrf, title: "too early" }), redirect: "manual" });
25192524 assert.equal(webWrite.status, 423, await webWrite.text());
modifiedcontainers/runner/src/importer.ts+7−6View file
@@ -395,14 +395,15 @@
395395 .map((l) => l.split(" "))
396396 .filter((p) => p[1] === "blob")
397397 .map((p) => ({ oid: p[0]!, size: Number(p[2]) }));
398- const batches = batchBySize(blobs, this.budget);
399- for (const [i, batch] of batches.entries()) {
398+ // Batched at the budget as it stands: when the server refuses a batch the budget shrinks, and
399+ // what is left is batched again at the new one. A lone object is retried as is (it can't be
400+ // split further), until the budget can't shrink any more.
401+ for (let left = blobs, n = 1; left.length; n++) {
402+ const batch = batchBySize(left, this.budget)[0]!;
400403 const tree = (await this.git.run(["mktree"], { input: batch.map((b, j) => `100644 blob ${b.oid}\tobject-${j}\n`).join("") })).stdout.trim();
401- const carrier = (await this.git.run(["commit-tree", tree, "-m", `Diffinery import: objects for ${commit.slice(0, 12)} (${i + 1}/${batches.length})`])).stdout.trim();
404+ const carrier = (await this.git.run(["commit-tree", tree, "-m", `Diffinery import: objects for ${commit.slice(0, 12)} (${n})`])).stdout.trim();
402405 const size = batch.reduce((s, b) => s + b.size, 0);
403- while (!(await this.pushTemp(carrier, size, label))) {
404- /* budget shrank; the carrier is retried as is (a lone object can't be split further) */
405- }
406+ if (await this.pushTemp(carrier, size, label)) left = left.slice(batch.length);
406407 }
407408 }
408409
modifiedcontainers/runner/test/import.test.ts+39−0View file
@@ -148,4 +148,43 @@
148148 expect(log.every((l) => l.endsWith("diffinery-import"))).toBe(true);
149149 expect(log.some((l) => Number(l.split(" ")[0]) > 153600)).toBe(true);
150150 });
151+ it("splits a refused batch of a large commit's files again, rather than retrying it whole", async () => {
152+ const root = mkdtempSync(path.join(tmpdir(), "diffinery-import-"));
153+ const work = path.join(root, "work");
154+ g(root, "init", "-q", "-b", "main", work);
155+ // One commit over the budget, of files that each fit in a push the server takes, but not all together.
156+ for (let i = 0; i < 8; i++) writeFileSync(path.join(work, `f${i}.bin`), randomBytes(40 * 1024));
157+ g(work, "add", "-A");
158+ g(work, "commit", "-q", "-m", "many files");
159+ const source = path.join(root, "source.git");
160+ g(root, "clone", "-q", "--bare", work, source);
161+ const target = path.join(root, "target.git");
162+ g(root, "init", "-q", "--bare", target);
163+ g(target, "config", "receive.advertisePushOptions", "true");
164+ const hook = path.join(target, "hooks", "pre-receive");
165+ writeFileSync(
166+ hook,
167+ `#!/bin/sh
168+size=$(du -sb "$GIT_QUARANTINE_PATH" 2>/dev/null | cut -f1)
169+if [ "\${size:-0}" -gt 153600 ]; then echo "pack takes more than 500 reads (or 1 bytes) of storage to unpack; split it into smaller pushes"; exit 1; fi
170+`,
171+ );
172+ chmodSync(hook, 0o755);
173+ const spec: ImportSpec = {
174+ importId: "t2",
175+ source: { fullName: "octo/many", cloneUrl: `file://${source}`, defaultBranch: "main" },
176+ githubToken: null,
177+ target: { fullName: "me/many", pushUrl: `file://${target}` },
178+ pushToken: "dfy_test",
179+ lfs: false,
180+ secrets: "bypass",
181+ bypassReason: "Imported from GitHub octo/many",
182+ // The commit is batched at 256 KB (six files), which the server refuses.
183+ chunkBytes: 256 * 1024,
184+ };
185+ const result = await transfer(spec, { workRoot: path.join(root, "agent"), progress: async () => {}, log: () => {} });
186+ expect(result.ok).toBe(true);
187+ expect(g(target, "rev-parse", "refs/heads/main")).toBe(g(work, "rev-parse", "HEAD"));
188+ g(target, "fsck", "--no-progress");
189+ });
151190 });