Best of N: a task can run up to 4 attempts side by side, then you pick one (phase 3d) (#108)
Claude-Session: https://claude.ai/code/session_014VDQqT8oL18uLJ9fpsNxW2 Co-authored-by: Claude <noreply@anthropic.com>
Canberk Demirci authored · GitHub committed signed1 parent 2abccfdcommit 6dc2ff42b508b73d0ce693d380aaab68aa9fa1f5Browse files
8 files changed+449−41
Files (8)
- apps/diffinery/docs/agents/tasks.md+15−3
- apps/diffinery/src/agentic/ProjectDO.ts+159−23
- apps/diffinery/src/agentic/api.ts+15−1
- apps/diffinery/src/agentic/project.ts+11−2
- apps/diffinery/src/agentic/tasks-web.tsx+85−7
- apps/diffinery/src/agentic/tasks.ts+56−4
- apps/diffinery/test/e2e/tasks.test.mjs+98−0
- apps/diffinery/test/unit/project.test.ts+10−1
modifiedapps/diffinery/docs/agents/tasks.md+15−3View file
| @@ -24,7 +24,8 @@ | ||
| 24 | 24 | 3. Pick the **runtime**: Claude Code, Codex or **Custom command** (see [Runtimes](#runtimes)). |
| 25 | 25 | 4. Pick the **agent** and the **base branch**. |
| 26 | 26 | 5. Tick **Plan first** to have the agent plan before it builds (see [Plan first](#plan-first)). |
| 27 | -6. Click **Start task**. | |
| 27 | +6. Choose **Attempts** to have up to 4 agents try the goal at once, then pick the best (see [Best of N](#best-of-n)). | |
| 28 | +7. Click **Start task**. | |
| 28 | 29 | |
| 29 | 30 | The task page shows the transcript live: what the agent says, the commands it runs and their output. |
| 30 | 31 | |
| @@ -51,7 +52,7 @@ | ||
| 51 | 52 | | **Failed** | The last run failed: the runtime exited with an error, the push was refused, or you lost access. The transcript says why. | |
| 52 | 53 | | **Stopped** | Someone stopped it. | |
| 53 | 54 | |
| 54 | -A [plan-first](#plan-first) task also has a plan state: `planning` while its plan runs, `waiting` when the plan waits for you (the board shows it under **Waiting on you**), then `approved` or `split`. | |
| 55 | +A [plan-first](#plan-first) task also has a plan state: `planning` while its plan runs, `waiting` when the plan waits for you (the board shows it under **Waiting on you**), then `approved` or `split`. A task with attempts has a pick state: `running`, `waiting` for your pick (also under **Waiting on you**), then `picked`, with `picked_attempt`. | |
| 55 | 56 | |
| 56 | 57 | ### Plan first |
| 57 | 58 | |
| @@ -63,6 +64,14 @@ | ||
| 63 | 64 | |
| 64 | 65 | Repository admins can make plans the default, or require them for every task, including ones coordinators start (see [Plan first](/docs/agents/projects#plan-first)). |
| 65 | 66 | |
| 67 | +### Best of N | |
| 68 | + | |
| 69 | +With **Attempts** set to 2, 3 or 4, the task's first build runs as that many attempts at the same time. Each one gets its own sandbox and pushes its own branch: the first pushes to the task's branch and the others to `-a2`, `-a3` and `-a4` beside it. Together they count as that many toward [How many at once](/docs/agents/projects#how-many-at-once), and each one spends your model key as its own run. With **Plan first**, the attempts build the approved plan. | |
| 70 | + | |
| 71 | +When the last attempt ends, the task waits on you with a comparison. For each attempt it shows the files and lines it changed (with a link to its diff), the checks on its branch, the tokens it used and its last message. **Pick** one, and its branch becomes the task's branch, with a pull request from it. The other attempts' branches are deleted, and every later run builds on the attempt you picked. Follow-ups wait for the pick. **Retry** before a pick runs all the attempts again, and **Stop** stops all of them. | |
| 72 | + | |
| 73 | +A task on a pull request's branch (**Fix it**) runs one attempt. | |
| 74 | + | |
| 66 | 75 | ### Steer, retry and stop |
| 67 | 76 | |
| 68 | 77 | - **Send a follow-up** from the task page to continue: a new run starts on the task's branch with the goal, how the previous run ended and your message, and its work goes to the same pull request. Sent while a run is active, the message waits and starts when that run ends. |
| @@ -149,9 +158,11 @@ | ||
| 149 | 158 | | `GET /repos/{owner}/{repo}/diffinery/tasks` | List tasks | Write access; any token | |
| 150 | 159 | | `GET /repos/{owner}/{repo}/diffinery/tasks/{number}` | A task with its runs | Write access; any token | |
| 151 | 160 | | `GET /repos/{owner}/{repo}/diffinery/tasks/{number}/transcript?after=0` | Transcript lines | Write access; any token | |
| 152 | -| `POST /repos/{owner}/{repo}/diffinery/tasks` | Start a task; `"plan_first": true` or `false` overrides the project's default, unless it requires plans | Browser session only | | |
| 161 | +| `POST /repos/{owner}/{repo}/diffinery/tasks` | Start a task; `"plan_first": true` or `false` overrides the project's default, unless it requires plans; `"attempts": 1` to `4` | Browser session only | | |
| 153 | 162 | | `POST .../tasks/{number}/messages` | Send a follow-up | Browser session only, the person who started it | |
| 154 | 163 | | `POST .../tasks/{number}/retry` | Retry | Browser session only, the person who started it | |
| 164 | +| `GET .../tasks/{number}/attempts` | The attempts compared: state, branch, pushed head, changes, checks, tokens, last message | Write access; any token | | |
| 165 | +| `POST .../tasks/{number}/attempts/{attempt}/pick` | Pick an attempt: its branch becomes the task's, with a pull request | Browser session only, the person who started it | | |
| 155 | 166 | | `POST .../tasks/{number}/plan/approve` | Approve a waiting plan: the build run starts | Browser session only, the person who started it | |
| 156 | 167 | | `POST .../tasks/{number}/plan/split` | Hand a waiting plan to your coordinator to split into tasks | Browser session only, the person who started it | |
| 157 | 168 | | `POST .../tasks/{number}/stop` | Stop | The person who started it, its agent or a repository admin | |
| @@ -174,6 +185,7 @@ | ||
| 174 | 185 | "agent": { "login": "ayse-bot", "type": "Bot" }, |
| 175 | 186 | "started_by": { "login": "ayse", "type": "User" }, |
| 176 | 187 | "plan_first": false, |
| 188 | + "attempts": 1, | |
| 177 | 189 | "base": "main", |
| 178 | 190 | "branch": "agent/7-fix-the-contact-sales-link-on-the", |
| 179 | 191 | "pull_request": { "number": 42, "html_url": "https://diffinery.com/acme/website/pull/42" }, |
modifiedapps/diffinery/src/agentic/ProjectDO.ts+159−23View file
| @@ -7,7 +7,7 @@ | ||
| 7 | 7 | import { PROVIDERS, type Provider } from "./keys.js"; |
| 8 | 8 | import { enqueueMail } from "../mail/send.js"; |
| 9 | 9 | import { formatTokens, pinRefusal, weighted, type Usage } from "./usage.js"; |
| 10 | -import { isPlanFirst, lineDiff, MAX_PLAN, MAX_POST, MAX_RUNNING_DEFAULT, MEMORY_VERSIONS, projectContext, type MemoryRow, type PlanState, type PostRow, type ProjectSettings } from "./project.js"; | |
| 10 | +import { attemptBranch, isPlanFirst, lineDiff, MAX_ATTEMPTS, MAX_PLAN, MAX_POST, MAX_RUNNING_DEFAULT, MEMORY_VERSIONS, projectContext, type MemoryRow, type PickState, type PlanState, type PostRow, type ProjectSettings } from "./project.js"; | |
| 11 | 11 | import { COORDINATOR_MAX_MODEL_CALLS, COORDINATOR_TIMEOUT_MS, COORDINATOR_TOKENS_DEFAULT, coordinatorPrompt, parseActions, TASKS_PER_DAY, TASKS_PER_RUN, type CoordinatorSettings } from "./coordinator.js"; |
| 12 | 12 | import { countText, dayAgo, findingComment, rulesFiles, findingId, MAX_AUTOFIXES, placeFindings, REPO_DAILY_DEFAULT, REVIEW_APP, REVIEW_CHECK, REVIEW_CREATOR, REVIEW_MAX_MODEL_CALLS, REVIEW_TIMEOUT_MS, REVIEW_TOKENS_DEFAULT, reviewBody, reviewLimit, reviewOutput, reviewPrompt, fixGoal, shouldMailLimit, type Finding, type PriorFinding, type ReviewLimit, type ReviewSettings } from "./review.js"; |
| 13 | 13 | import { agentEnv, finalMessage, INSTALL_TIMEOUT_MS, planPrompt, RUN_TIMEOUT_MS, RUNTIMES, runPrompt, runtimeLabel, taskBranch, taskTitle, type ModelRoute, type RuntimeId, type TranscriptLine } from "./runtimes.js"; |
| @@ -84,6 +84,11 @@ | ||
| 84 | 84 | plan: string | null; |
| 85 | 85 | /** A task a coordinator started while splitting another task's plan: that task. */ |
| 86 | 86 | parent_task: number | null; |
| 87 | + /** Best of N (phase 3d): how many attempts its first build runs as, each in its own sandbox on its own branch. */ | |
| 88 | + attempts: number; | |
| 89 | + /** With more than one attempt: they run, then wait for the person to pick one, which becomes the task's branch. */ | |
| 90 | + pick_state: PickState | null; | |
| 91 | + picked: number | null; | |
| 87 | 92 | created_at: string; |
| 88 | 93 | updated_at: string; |
| 89 | 94 | }; |
| @@ -118,12 +123,30 @@ | ||
| 118 | 123 | checked_at: string | null; |
| 119 | 124 | /** A coordinator run: when it began applying its actions, so a retried alarm never applies them twice. */ |
| 120 | 125 | applied_at: string | null; |
| 126 | + /** One of a task's attempts (best of N): which, and the branch it pushes to. */ | |
| 127 | + attempt: number | null; | |
| 128 | + branch: string | null; | |
| 121 | 129 | started_at: string; |
| 122 | 130 | ended_at: string | null; |
| 123 | 131 | }; |
| 124 | 132 | |
| 133 | +/** One attempt in a task's comparison (best of N). */ | |
| 134 | +export interface AttemptView { | |
| 135 | + attempt: number; | |
| 136 | + run: number; | |
| 137 | + step: RunStep; | |
| 138 | + error: string | null; | |
| 139 | + branch: string; | |
| 140 | + start_sha: string | null; | |
| 141 | + /** What it pushed; null when it changed nothing or hasn't finished. */ | |
| 142 | + head_sha: string | null; | |
| 143 | + model_calls: number; | |
| 144 | + tokens: number; | |
| 145 | + summary: string | null; | |
| 146 | +} | |
| 147 | + | |
| 125 | 148 | export interface TaskView extends TaskRow { |
| 126 | - runs: Array<Pick<RunRow, "seq" | "kind" | "message" | "step" | "exit_code" | "error" | "model_calls" | "tokens" | "started_at" | "ended_at">>; | |
| 149 | + runs: Array<Pick<RunRow, "seq" | "kind" | "message" | "step" | "exit_code" | "error" | "model_calls" | "tokens" | "started_at" | "ended_at" | "attempt">>; | |
| 127 | 150 | } |
| 128 | 151 | |
| 129 | 152 | const SCHEMA = ` |
| @@ -255,6 +278,11 @@ | ||
| 255 | 278 | ["tasks", "plan", "plan TEXT"], |
| 256 | 279 | ["tasks", "parent_task", "parent_task INTEGER"], |
| 257 | 280 | ["posts", "split_of", "split_of INTEGER"], |
| 281 | + ["tasks", "attempts", "attempts INTEGER NOT NULL DEFAULT 1"], | |
| 282 | + ["tasks", "pick_state", "pick_state TEXT"], | |
| 283 | + ["tasks", "picked", "picked INTEGER"], | |
| 284 | + ["runs", "attempt", "attempt INTEGER"], | |
| 285 | + ["runs", "branch", "branch TEXT"], | |
| 258 | 286 | ]; |
| 259 | 287 | |
| 260 | 288 | /** A finding as stored. */ |
| @@ -286,6 +314,8 @@ | ||
| 286 | 314 | /** How much of the base branch's REVIEW.md / BUGBOT.md files goes into a review's prompt. */ |
| 287 | 315 | const MAX_RULES_BYTES = 48 * 1024; |
| 288 | 316 | const MAX_LINES_PER_RUN = 20_000; |
| 317 | +/** How much of each attempt's final message the comparison shows. */ | |
| 318 | +const MAX_SUMMARY = 4_000; | |
| 289 | 319 | |
| 290 | 320 | export interface CreateTask { |
| 291 | 321 | repoId: string; |
| @@ -307,6 +337,8 @@ | ||
| 307 | 337 | coordinator?: { post: number; tokenBudget: number; after: number[]; parent?: number | null }; |
| 308 | 338 | /** Plan first (agentic/project.ts): the first run writes a plan for the person to approve. */ |
| 309 | 339 | planFirst?: boolean; |
| 340 | + /** Best of N: the first build runs as this many attempts (1 to MAX_ATTEMPTS). */ | |
| 341 | + attempts?: number; | |
| 310 | 342 | } |
| 311 | 343 | |
| 312 | 344 | export interface ReviewRequest { |
| @@ -346,7 +378,7 @@ | ||
| 346 | 378 | get(number: number): TaskView | null { |
| 347 | 379 | const t = this.task(number); |
| 348 | 380 | if (!t) return null; |
| 349 | - const runs = this.sql.exec<RunRow>("SELECT seq, kind, message, step, exit_code, error, model_calls, tokens, started_at, ended_at FROM runs WHERE task = ? ORDER BY seq", number).toArray(); | |
| 381 | + const runs = this.sql.exec<RunRow>("SELECT seq, kind, message, step, exit_code, error, model_calls, tokens, started_at, ended_at, attempt FROM runs WHERE task = ? ORDER BY seq", number).toArray(); | |
| 350 | 382 | return { ...t, runs }; |
| 351 | 383 | } |
| 352 | 384 | |
| @@ -484,7 +516,7 @@ | ||
| 484 | 516 | const branch = input.pull?.head ?? taskBranch(number, input.title || input.goal); |
| 485 | 517 | const title = input.title ? taskTitle(input.title) : taskTitle(input.goal); |
| 486 | 518 | this.sql.exec( |
| 487 | - "INSERT INTO tasks (number, id, repo_id, goal, title, runtime, command, via, model, agent_id, agent_login, person_id, person_login, base, branch, state, pr_number, autofix_of, coordinator_post, token_budget, after_tasks, plan_first, plan_state, parent_task, created_at, updated_at) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, 'running', ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)", | |
| 519 | + "INSERT INTO tasks (number, id, repo_id, goal, title, runtime, command, via, model, agent_id, agent_login, person_id, person_login, base, branch, state, pr_number, autofix_of, coordinator_post, token_budget, after_tasks, plan_first, plan_state, parent_task, attempts, created_at, updated_at) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, 'running', ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)", | |
| 488 | 520 | number, |
| 489 | 521 | crypto.randomUUID(), |
| 490 | 522 | input.repoId, |
| @@ -508,21 +540,40 @@ | ||
| 508 | 540 | input.planFirst ? 1 : 0, |
| 509 | 541 | input.planFirst ? "planning" : null, |
| 510 | 542 | input.coordinator?.parent ?? null, |
| 543 | + Math.min(Math.max(input.attempts ?? 1, 1), MAX_ATTEMPTS), | |
| 511 | 544 | at, |
| 512 | 545 | at, |
| 513 | 546 | ); |
| 514 | 547 | this.event(number, `${input.autofixOf ? "Autofix" : input.coordinator ? `${input.person.login}'s coordinator` : input.person.login} started task #${number}: ${title}`); |
| 515 | - this.startRun(number, input.planFirst ? "plan" : "start", null, input.providers); | |
| 548 | + if (input.planFirst) this.startRun(number, "plan", null, input.providers); | |
| 549 | + else this.startAttempts(number, "start", input.providers); | |
| 516 | 550 | await this.index(number); |
| 517 | 551 | return this.get(number)!; |
| 518 | 552 | } |
| 519 | 553 | |
| 520 | 554 | /** |
| 555 | + * The first build of a task: one run, or with best of N one run per attempt, each on its own branch, started | |
| 556 | + * together. Together they count as N toward the project's "how many at once". | |
| 557 | + */ | |
| 558 | + private startAttempts(number: number, kind: "start" | "build" | "retry", providers: Provider[]): void { | |
| 559 | + const t = this.mustTask(number); | |
| 560 | + if (t.attempts <= 1) return this.startRun(number, kind, null, providers); | |
| 561 | + this.sql.exec("UPDATE tasks SET pick_state = 'running', picked = NULL WHERE number = ?", number); | |
| 562 | + for (let k = 1; k <= t.attempts; k++) this.startRun(number, kind, null, providers, k); | |
| 563 | + } | |
| 564 | + | |
| 565 | + /** Best of N before a pick: the attempts are running, or wait for the person to pick one. */ | |
| 566 | + private picking(t: TaskRow): boolean { | |
| 567 | + return t.attempts > 1 && (t.pick_state === "running" || t.pick_state === "waiting"); | |
| 568 | + } | |
| 569 | + | |
| 570 | + /** | |
| 521 | 571 | * A follow-up from the person who started the task. While a run is active it |
| 522 | 572 | * waits, and starts as soon as that run ends. |
| 523 | 573 | */ |
| 524 | 574 | async message(number: number, text: string, providers: Provider[], from?: string): Promise<TaskView> { |
| 525 | 575 | const t = this.mustTask(number); |
| 576 | + if (this.picking(t)) throw new HttpError(409, t.pick_state === "running" ? `the task is running ${t.attempts} attempts: send follow-ups once you've picked one` : "pick an attempt first: follow-ups go to the picked attempt's branch"); | |
| 526 | 577 | if (this.activeRun(number)) { |
| 527 | 578 | this.sql.exec("UPDATE tasks SET queued = ?, updated_at = ? WHERE number = ?", t.queued ? `${t.queued}\n\n${text}` : text, now(), number); |
| 528 | 579 | this.note(number, this.lastRun(number)?.seq ?? 0, `${from ?? t.person_login} sent a message; it starts a new run when this one ends.`); |
| @@ -535,10 +586,12 @@ | ||
| 535 | 586 | } |
| 536 | 587 | |
| 537 | 588 | async retry(number: number, providers: Provider[]): Promise<TaskView> { |
| 538 | - this.mustTask(number); | |
| 589 | + const t = this.mustTask(number); | |
| 539 | 590 | if (this.activeRun(number)) throw new HttpError(409, "the task is running; stop it before retrying"); |
| 540 | 591 | const last = this.lastRun(number); |
| 541 | - this.startRun(number, last?.kind === "plan" ? "plan" : "retry", last?.message ?? null, providers); | |
| 592 | + // Attempts that weren't picked run again, all of them. | |
| 593 | + if (last?.attempt && t.picked === null) this.startAttempts(number, "retry", providers); | |
| 594 | + else this.startRun(number, last?.kind === "plan" ? "plan" : "retry", last?.message ?? null, providers); | |
| 542 | 595 | await this.index(number); |
| 543 | 596 | return this.get(number)!; |
| 544 | 597 | } |
| @@ -546,10 +599,11 @@ | ||
| 546 | 599 | /** Stop the active run: kill the agent, close the sandbox (revoking its git credential) and the model token. */ |
| 547 | 600 | async stop(number: number, by: string, reason?: string): Promise<TaskView> { |
| 548 | 601 | this.mustTask(number); |
| 549 | - const run = this.activeRun(number); | |
| 602 | + const runs = this.sql.exec<RunRow>(`SELECT * FROM runs WHERE task = ? AND step IN (${ACTIVE.map(() => "?").join(", ")}) ORDER BY seq`, number, ...ACTIVE).toArray(); | |
| 550 | 603 | this.sql.exec("UPDATE tasks SET queued = NULL WHERE number = ?", number); |
| 551 | - if (run) await this.endRun(run, "stopped", reason ?? `Stopped by ${by}.`); | |
| 552 | - else this.event(number, `${by} stopped task #${number}.`); | |
| 604 | + // Every attempt that is still running stops. | |
| 605 | + for (const run of runs) await this.endRun(run, "stopped", reason ?? `Stopped by ${by}.`); | |
| 606 | + if (!runs.length) this.event(number, `${by} stopped task #${number}.`); | |
| 553 | 607 | this.sql.exec("UPDATE tasks SET state = 'stopped', updated_at = ? WHERE number = ?", now(), number); |
| 554 | 608 | await this.index(number); |
| 555 | 609 | return this.get(number)!; |
| @@ -562,7 +616,7 @@ | ||
| 562 | 616 | const t = this.waitingPlan(number); |
| 563 | 617 | this.sql.exec("UPDATE tasks SET plan_state = 'approved' WHERE number = ?", number); |
| 564 | 618 | this.event(number, `${by} approved the plan of task #${number}.`); |
| 565 | - this.startRun(t.number, "build", null, providers); | |
| 619 | + this.startAttempts(t.number, "build", providers); | |
| 566 | 620 | await this.index(number); |
| 567 | 621 | return this.get(number)!; |
| 568 | 622 | } |
| @@ -582,6 +636,56 @@ | ||
| 582 | 636 | return { post: p.n, ...out }; |
| 583 | 637 | } |
| 584 | 638 | |
| 639 | + // ---------------------------------------------------------------- best of N | |
| 640 | + | |
| 641 | + /** The comparison of a task's latest attempts: each one's state, branch, pushed head, usage and final message. */ | |
| 642 | + attempts(number: number): AttemptView[] { | |
| 643 | + const t = this.mustTask(number); | |
| 644 | + return this.attemptRuns(number).map((r) => ({ attempt: r.attempt!, run: r.seq, step: r.step, error: r.error, branch: r.branch!, start_sha: r.start_sha, head_sha: r.head_sha, model_calls: r.model_calls ?? 0, tokens: r.tokens ?? 0, summary: this.attemptSummary(t, r) })); | |
| 645 | + } | |
| 646 | + | |
| 647 | + /** A task's latest attempts: the newest batch only. */ | |
| 648 | + private attemptRuns(number: number): RunRow[] { | |
| 649 | + const t = this.mustTask(number); | |
| 650 | + if (t.attempts <= 1) return []; | |
| 651 | + const runs = this.sql.exec<RunRow>("SELECT * FROM runs WHERE task = ? AND attempt IS NOT NULL ORDER BY seq DESC LIMIT ?", number, t.attempts).toArray(); | |
| 652 | + return runs.reverse(); | |
| 653 | + } | |
| 654 | + | |
| 655 | + /** What each attempt said last: its final message (a custom command's last lines). */ | |
| 656 | + private attemptSummary(t: TaskRow, r: RunRow): string | null { | |
| 657 | + const lines = this.sql.exec<TranscriptLine>("SELECT seq, run, at, stream, text FROM lines WHERE task = ? AND run = ? AND stream = 'out' ORDER BY seq DESC LIMIT 200", t.number, r.seq).toArray().reverse(); | |
| 658 | + const text = t.runtime === "custom" ? lines.slice(-20).map((l) => l.text).join("\n") : finalMessage(lines); | |
| 659 | + return text?.trim() ? (text.length > MAX_SUMMARY ? `${text.slice(0, MAX_SUMMARY)} … (cut)` : text.trim()) : null; | |
| 660 | + } | |
| 661 | + | |
| 662 | + /** | |
| 663 | + * Pick one attempt: its branch becomes the task's branch, with a pull request from it, and every later run builds | |
| 664 | + * on it. The other attempts' branches are deleted, as the agent that pushed them. | |
| 665 | + */ | |
| 666 | + async pick(number: number, attempt: number, by: string): Promise<TaskView> { | |
| 667 | + const t = this.mustTask(number); | |
| 668 | + if (t.pick_state !== "waiting") throw new HttpError(409, t.pick_state === "running" ? "the attempts are still running" : t.pick_state === "picked" ? `attempt ${t.picked} is already picked` : "this task has no attempts to pick from"); | |
| 669 | + if (this.activeRun(number)) throw new HttpError(409, "the task is running"); | |
| 670 | + const runs = this.attemptRuns(number); | |
| 671 | + const r = runs.find((x) => x.attempt === attempt); | |
| 672 | + if (!r) throw new HttpError(404, "no such attempt"); | |
| 673 | + if (r.step !== "done" || !r.head_sha || !r.branch) throw new HttpError(422, `attempt ${attempt} has no work to pick: it ${r.step === "done" ? "changed nothing" : r.step}`); | |
| 674 | + this.sql.exec("UPDATE tasks SET branch = ?, picked = ?, pick_state = 'picked', updated_at = ? WHERE number = ?", r.branch, attempt, now(), number); | |
| 675 | + this.event(number, `${by} picked attempt ${attempt} of task #${number}.`); | |
| 676 | + this.note(number, r.seq, `${by} picked this attempt: ${r.branch} is now the task's branch.`); | |
| 677 | + const picked = this.mustTask(number); | |
| 678 | + await this.openPull(picked, r); | |
| 679 | + const { caller } = await this.checkAccess(picked); | |
| 680 | + const stub = this.env.REPO.get(this.env.REPO.idFromName(t.repo_id)); | |
| 681 | + for (const o of runs) { | |
| 682 | + if (o.attempt === attempt || !o.branch || o.branch === r.branch) continue; | |
| 683 | + if (await stub.getRef(`refs/heads/${o.branch}`)) await stub.deleteRef(`refs/heads/${o.branch}`, caller).catch((e) => this.note(number, o.seq, `Couldn't delete ${o.branch}: ${e instanceof Error ? e.message : String(e)}`)); | |
| 684 | + } | |
| 685 | + await this.index(number); | |
| 686 | + return this.get(number)!; | |
| 687 | + } | |
| 688 | + | |
| 585 | 689 | private waitingPlan(number: number): TaskRow { |
| 586 | 690 | const t = this.mustTask(number); |
| 587 | 691 | if (t.plan_state !== "waiting" || !t.plan) throw new HttpError(409, t.plan_state === "planning" ? "the plan isn't ready yet" : "this task has no plan waiting for approval"); |
| @@ -777,10 +881,12 @@ | ||
| 777 | 881 | |
| 778 | 882 | // ---------------------------------------------------------------- runs |
| 779 | 883 | |
| 780 | - private startRun(number: number, kind: RunRow["kind"], message: string | null, providers: Provider[]): void { | |
| 884 | + private startRun(number: number, kind: RunRow["kind"], message: string | null, providers: Provider[], attempt: number | null = null): void { | |
| 781 | 885 | const t = this.mustTask(number); |
| 782 | 886 | const seq = (this.lastRun(number)?.seq ?? 0) + 1; |
| 783 | - const previous = seq > 1 && t.kind === "task" ? finalMessage(this.sql.exec<TranscriptLine>("SELECT seq, run, at, stream, text FROM lines WHERE task = ? AND run = ? ORDER BY seq DESC LIMIT 200", number, seq - 1).toArray().reverse()) : null; | |
| 887 | + // A follow-up continues from the run before it on the task's branch: after a pick, the picked attempt's. | |
| 888 | + const prior = seq > 1 && t.kind === "task" ? this.sql.exec<{ seq: number }>("SELECT seq FROM runs WHERE task = ? AND (attempt IS NULL OR attempt = ?) ORDER BY seq DESC LIMIT 1", number, t.picked ?? 0).toArray()[0]?.seq : undefined; | |
| 889 | + const previous = prior ? finalMessage(this.sql.exec<TranscriptLine>("SELECT seq, run, at, stream, text FROM lines WHERE task = ? AND run = ? ORDER BY seq DESC LIMIT 200", number, prior).toArray().reverse()) : null; | |
| 784 | 890 | // Reviews' and coordinators' prompts are built when they boot, from the state then. Reviews never carry |
| 785 | 891 | // project memory (a pull request mustn't be reviewed against notes an agent wrote); they get the instructions. |
| 786 | 892 | const own = t.kind !== "task"; |
| @@ -794,14 +900,17 @@ | ||
| 794 | 900 | const at = now(); |
| 795 | 901 | // A task's run waits its turn under the project's "how many at once"; reviews and coordinators have their own limits. |
| 796 | 902 | const step: RunStep = own ? "boot" : "queued"; |
| 797 | - this.sql.exec("INSERT INTO runs (task, seq, kind, message, prompt, providers, step, started_at) VALUES (?, ?, ?, ?, ?, ?, ?, ?)", number, seq, kind, message, prompt, JSON.stringify(providers), step, at); | |
| 798 | - this.sql.exec("UPDATE tasks SET state = ?, error = NULL, updated_at = ? WHERE number = ?", step === "queued" ? "queued" : "running", at, number); | |
| 903 | + this.sql.exec("INSERT INTO runs (task, seq, kind, message, prompt, providers, step, started_at, attempt, branch) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)", number, seq, kind, message, prompt, JSON.stringify(providers), step, at, attempt, attempt ? attemptBranch(t.branch, attempt) : null); | |
| 904 | + // A task with an attempt already running stays running while the next one queues. | |
| 905 | + const busy = this.sql.exec<{ n: number }>(`SELECT count(*) AS n FROM runs WHERE task = ? AND step IN (${RUNNING.map(() => "?").join(", ")})`, number, ...RUNNING).one().n > 0; | |
| 906 | + this.sql.exec("UPDATE tasks SET state = ?, error = NULL, updated_at = ? WHERE number = ?", step === "queued" && !busy ? "queued" : "running", at, number); | |
| 799 | 907 | if (kind === "review") this.note(number, seq, `${message ? `${message} asked for a review` : "Reviewing automatically"}, as ${t.agent_login} with ${runtimeLabel(t.runtime, t)} and ${t.person_login}'s model key.`); |
| 800 | 908 | else if (kind === "coordinate") { |
| 801 | 909 | const posts = JSON.parse(message ?? "[]") as number[]; |
| 802 | 910 | this.note(number, seq, `Handling ${posts.map((p) => `post #${p}`).join(", ")} as ${t.agent_login} with ${runtimeLabel(t.runtime, t)} and ${t.person_login}'s model key.`); |
| 803 | 911 | } else { |
| 804 | 912 | if (message) this.line(number, seq, "note", `${t.person_login}: ${message}`); |
| 913 | + if (attempt) this.note(number, seq, `Attempt ${attempt} of ${t.attempts}, on ${attemptBranch(t.branch, attempt)}.`); | |
| 805 | 914 | this.note( |
| 806 | 915 | number, |
| 807 | 916 | seq, |
| @@ -938,17 +1047,23 @@ | ||
| 938 | 1047 | if (t.kind === "review") return this.collect(t, r); |
| 939 | 1048 | if (t.kind === "coordinator") return this.coordinate(t, r); |
| 940 | 1049 | if (r.kind === "plan") return this.keepPlan(t, r); |
| 1050 | + const branch = r.branch ?? t.branch; | |
| 941 | 1051 | if (!r.proc) { |
| 942 | - return this.startProc(r, "finish", FINISH_SCRIPT, { DIFFINERY_TASK_BRANCH: t.branch, DIFFINERY_TASK_START: r.start_sha ?? "", DIFFINERY_TASK_MESSAGE: commitMessage(t, r) }, {}, 10 * 60 * 1000, `Pushing to ${t.branch}.`); | |
| 1052 | + return this.startProc(r, "finish", FINISH_SCRIPT, { DIFFINERY_TASK_BRANCH: branch, DIFFINERY_TASK_START: r.start_sha ?? "", DIFFINERY_TASK_MESSAGE: commitMessage(t, r) }, {}, 10 * 60 * 1000, `Pushing to ${branch}.`); | |
| 943 | 1053 | } |
| 944 | 1054 | const done = await this.poll(r, "setup"); |
| 945 | 1055 | if (done === null) return; |
| 946 | 1056 | const run = this.run(r.task, r.seq)!; |
| 947 | - if (done !== 0) return this.endRun(run, "failed", `Pushing the work to ${t.branch} failed (exit ${done}).`); | |
| 1057 | + if (done !== 0) return this.endRun(run, "failed", `Pushing the work to ${branch} failed (exit ${done}).`); | |
| 948 | 1058 | // Whether anything was pushed comes from the branch itself, not from output the agent could have written to. |
| 949 | - const head = await this.env.REPO.get(this.env.REPO.idFromName(t.repo_id)).getRef(`refs/heads/${t.branch}`); | |
| 950 | - if (head && head !== r.start_sha) await this.openPull(t, run); | |
| 951 | - else this.note(r.task, r.seq, "No changes to push."); | |
| 1059 | + const head = await this.env.REPO.get(this.env.REPO.idFromName(t.repo_id)).getRef(`refs/heads/${branch}`); | |
| 1060 | + if (head && head !== r.start_sha) { | |
| 1061 | + // An attempt waits to be picked; its pull request opens then. | |
| 1062 | + if (r.attempt) { | |
| 1063 | + this.sql.exec("UPDATE runs SET head_sha = ? WHERE task = ? AND seq = ?", head, r.task, r.seq); | |
| 1064 | + this.note(r.task, r.seq, `Pushed attempt ${r.attempt} to ${branch}.`); | |
| 1065 | + } else await this.openPull(t, run); | |
| 1066 | + } else this.note(r.task, r.seq, "No changes to push."); | |
| 952 | 1067 | const fresh = this.run(r.task, r.seq)!; |
| 953 | 1068 | return this.endRun(fresh, (fresh.exit_code ?? 0) === 0 ? "done" : "failed", (fresh.exit_code ?? 0) === 0 ? null : `${RUNTIMES[t.runtime].name} exited with code ${fresh.exit_code}.`); |
| 954 | 1069 | } |
| @@ -970,8 +1085,9 @@ | ||
| 970 | 1085 | private async boot(t: TaskRow, r: RunRow): Promise<void> { |
| 971 | 1086 | const { caller, repo } = await this.checkAccess(t); |
| 972 | 1087 | const stub = this.env.REPO.get(this.env.REPO.idFromName(t.repo_id)); |
| 973 | - const branchSha = await stub.getRef(`refs/heads/${t.branch}`); | |
| 974 | - const ref = branchSha ? t.branch : t.base; | |
| 1088 | + const branch = r.branch ?? t.branch; | |
| 1089 | + const branchSha = await stub.getRef(`refs/heads/${branch}`); | |
| 1090 | + const ref = branchSha ? branch : t.base; | |
| 975 | 1091 | const startSha = branchSha ?? (await stub.getRef(`refs/heads/${t.base}`)); |
| 976 | 1092 | if (!startSha) throw new HttpError(422, `the branch ${t.base} doesn't exist in ${repo.owner_login}/${repo.name}`); |
| 977 | 1093 | // A plan run can't change code: its sandbox is read-only, as a review's. |
| @@ -1387,7 +1503,27 @@ | ||
| 1387 | 1503 | this.sql.exec("UPDATE tasks SET queued = NULL WHERE number = ?", r.task); |
| 1388 | 1504 | t.queued = null; |
| 1389 | 1505 | } |
| 1390 | - if (t.queued && step !== "stopped") { | |
| 1506 | + if (r.attempt && this.activeRun(r.task)) { | |
| 1507 | + // Other attempts are still running: the task's state waits for the last of them. | |
| 1508 | + } else if (r.attempt && t.pick_state === "running") { | |
| 1509 | + const runs = this.attemptRuns(r.task); | |
| 1510 | + const ready = runs.filter((x) => x.step === "done" && x.head_sha); | |
| 1511 | + const all = runs.length; | |
| 1512 | + if (step === "stopped" || runs.some((x) => x.step === "stopped")) { | |
| 1513 | + this.sql.exec("UPDATE tasks SET state = 'stopped', pick_state = NULL, error = ?, updated_at = ? WHERE number = ?", error, at, r.task); | |
| 1514 | + this.event(r.task, `Task #${r.task} stopped: ${error ?? "stopped"}`); | |
| 1515 | + } else if (ready.length) { | |
| 1516 | + this.sql.exec("UPDATE tasks SET state = 'ready', pick_state = 'waiting', error = NULL, updated_at = ? WHERE number = ?", at, r.task); | |
| 1517 | + this.event(r.task, `Task #${r.task}'s ${all} attempts are done: ${ready.length === all ? "all" : `${ready.length} of them`} pushed work to compare. Pick one.`); | |
| 1518 | + } else if (runs.every((x) => x.step === "failed")) { | |
| 1519 | + const why = `All ${all} attempts failed.`; | |
| 1520 | + this.sql.exec("UPDATE tasks SET state = 'failed', error = ?, updated_at = ? WHERE number = ?", why, at, r.task); | |
| 1521 | + this.event(r.task, `Task #${r.task} failed: ${why}`); | |
| 1522 | + } else { | |
| 1523 | + this.sql.exec("UPDATE tasks SET state = 'ready', pick_state = NULL, error = NULL, updated_at = ? WHERE number = ?", at, r.task); | |
| 1524 | + this.event(r.task, `Task #${r.task}'s ${all} attempts finished with no changes to review.`); | |
| 1525 | + } | |
| 1526 | + } else if (t.queued && step !== "stopped") { | |
| 1391 | 1527 | this.sql.exec("UPDATE tasks SET queued = NULL WHERE number = ?", r.task); |
| 1392 | 1528 | if (t.kind === "review") await this.restartReview(t, t.queued === "automatic" ? null : t.queued, JSON.parse(r.providers) as Provider[]); |
| 1393 | 1529 | else this.startRun(r.task, t.kind === "coordinator" ? "coordinate" : this.task(r.task)!.plan_state === "planning" || this.task(r.task)!.plan_state === "waiting" ? "plan" : "message", t.queued, JSON.parse(r.providers) as Provider[]); |
modifiedapps/diffinery/src/agentic/api.ts+15−1View file
| @@ -4,7 +4,7 @@ | ||
| 4 | 4 | import { loadRepo } from "../http/common.js"; |
| 5 | 5 | import { HttpError } from "../lib/util.js"; |
| 6 | 6 | import { isProvider, ModelKeys, type ModelKeyInfo } from "./keys.js"; |
| 7 | -import { approvePlan, getTask, lineJson, messageTask, myTasks, projectStub, requirePersonSession, requireTaskReader, retryTask, startTask, stopTask, taskJson, taskNumber, type StartInput } from "./tasks.js"; | |
| 7 | +import { approvePlan, attemptJson, compareAttempts, getTask, pickAttempt, lineJson, messageTask, myTasks, projectStub, requirePersonSession, requireTaskReader, retryTask, startTask, stopTask, taskJson, taskNumber, type StartInput } from "./tasks.js"; | |
| 8 | 8 | import { askForReview, findingJson, reviewSettings, reviewSettingsJson, saveReviewSettings, type SettingsInput } from "./reviews.js"; |
| 9 | 9 | import { gatherToday, todayJson } from "./today.js"; |
| 10 | 10 | import { boardJson, coordinatorJson, handlePost, splitPlan, loadBoard, memoryJson, postJson, postToProject, saveCoordinator, saveMemory, saveProjectSettings, settingsJson, type CoordinatorInput } from "./projects.js"; |
| @@ -177,6 +177,20 @@ | ||
| 177 | 177 | return c.json({ ...postJson(base(c), full(l), post), coordinator: handled }, 201); |
| 178 | 178 | }); |
| 179 | 179 | |
| 180 | +agenticApi.get("/repos/:owner/:repo/diffinery/tasks/:number/attempts", async (c) => { | |
| 181 | + const l = await repoOf(c); | |
| 182 | + const n = taskNumber(c.req.param("number")); | |
| 183 | + await getTask(c, l, n); | |
| 184 | + return c.json((await compareAttempts(c, l, n)).map((a) => attemptJson(base(c), full(l), a))); | |
| 185 | +}); | |
| 186 | + | |
| 187 | +agenticApi.post("/repos/:owner/:repo/diffinery/tasks/:number/attempts/:attempt/pick", async (c) => { | |
| 188 | + const l = await repoOf(c); | |
| 189 | + const k = Number(c.req.param("attempt")); | |
| 190 | + if (!Number.isInteger(k) || k < 1) throw new HttpError(404, "no such attempt"); | |
| 191 | + return c.json(taskJson(base(c), full(l), await pickAttempt(c, l, taskNumber(c.req.param("number")), k)), 202); | |
| 192 | +}); | |
| 193 | + | |
| 180 | 194 | agenticApi.post("/repos/:owner/:repo/diffinery/tasks/:number/plan/approve", async (c) => { |
| 181 | 195 | const l = await repoOf(c); |
| 182 | 196 | return c.json(taskJson(base(c), full(l), await approvePlan(c, l, taskNumber(c.req.param("number")))), 202); |
modifiedapps/diffinery/src/agentic/project.ts+11−2View file
| @@ -30,6 +30,13 @@ | ||
| 30 | 30 | /** The longest plan kept from a plan run's final message. */ |
| 31 | 31 | export const MAX_PLAN = 20_000; |
| 32 | 32 | |
| 33 | +/** Best of N: the most attempts a task may run at once. */ | |
| 34 | +export const MAX_ATTEMPTS = 4; | |
| 35 | +/** Where a task's attempts stand: running, waiting for the person to pick one, or picked. */ | |
| 36 | +export type PickState = "running" | "waiting" | "picked"; | |
| 37 | +/** Attempt 1 pushes to the task's branch, the others to `-a2`, `-a3`… beside it. */ | |
| 38 | +export const attemptBranch = (branch: string, attempt: number): string => (attempt === 1 ? branch : `${branch}-a${attempt}`); | |
| 39 | + | |
| 33 | 40 | /** Where a plan-first task stands: its plan runs, the plan waits for the person, then it is built or split. */ |
| 34 | 41 | export type PlanState = "planning" | "waiting" | "approved" | "split"; |
| 35 | 42 | |
| @@ -64,7 +71,7 @@ | ||
| 64 | 71 | export type BoardState = (typeof BOARD)[number]; |
| 65 | 72 | |
| 66 | 73 | export const BOARD_TEXT: Record<BoardState, { label: string; tone: string; hint: string }> = { |
| 67 | - waiting: { label: "Waiting on you", tone: "amber", hint: "Failed, a plan to approve, or its pull request has failing checks." }, | |
| 74 | + waiting: { label: "Waiting on you", tone: "amber", hint: "Failed, a plan to approve, attempts to pick from, or its pull request has failing checks." }, | |
| 68 | 75 | working: { label: "Working", tone: "blue", hint: "The agent is running, or its pull request's checks are." }, |
| 69 | 76 | queued: { label: "Queued", tone: "", hint: "Starts when a running task finishes." }, |
| 70 | 77 | review: { label: "Ready for review", tone: "green", hint: "Pull request open, checks green." }, |
| @@ -77,8 +84,10 @@ | ||
| 77 | 84 | export type PullState = { open: true; pull: Pick<TodayPull, "checks" | "approvedHead" | "autoMerge" | "queued" | "draft"> } | { open: false; merged: boolean }; |
| 78 | 85 | |
| 79 | 86 | /** Where a task stands on the board: its own state, then its pull request's. */ |
| 80 | -export function boardState(task: { state: string; pr_number: number | null; plan_state?: PlanState | null }, pull: PullState | null): BoardState { | |
| 87 | +export function boardState(task: { state: string; pr_number: number | null; plan_state?: PlanState | null; pick_state?: PickState | null }, pull: PullState | null): BoardState { | |
| 81 | 88 | if (task.state === "queued") return "queued"; |
| 89 | + // Attempts waiting for a pick wait on you. | |
| 90 | + if (task.state === "ready" && task.pick_state === "waiting") return "waiting"; | |
| 82 | 91 | // A plan waiting for approval waits on you; a plan split into other tasks is done with. |
| 83 | 92 | if (task.state === "ready" && task.plan_state === "waiting") return "waiting"; |
| 84 | 93 | if (task.state === "ready" && task.plan_state === "split") return "idle"; |
modifiedapps/diffinery/src/agentic/tasks-web.tsx+85−7View file
| @@ -10,8 +10,10 @@ | ||
| 10 | 10 | import { isProvider, ModelKeys, PROVIDERS, type Provider } from "./keys.js"; |
| 11 | 11 | import type { TaskRow, TaskView } from "./ProjectDO.js"; |
| 12 | 12 | import { fixGoal } from "./review.js"; |
| 13 | +import { MAX_ATTEMPTS } from "./project.js"; | |
| 14 | +import { formatTokens } from "./usage.js"; | |
| 13 | 15 | import { OPENROUTER_DEFAULT_MODEL, parseRuntimeChoice, runtimeChoices, runtimeLabel, transcriptItem, type TranscriptItem } from "./runtimes.js"; |
| 14 | -import { approvePlan, drivableAgents, getTask, mayStop, messageTask, projectStub, requireTaskReader, retryTask, startTask, stopTask, taskNumber } from "./tasks.js"; | |
| 16 | +import { approvePlan, compareAttempts, drivableAgents, getTask, pickAttempt, type AttemptCompare, mayStop, messageTask, projectStub, requireTaskReader, retryTask, startTask, stopTask, taskNumber } from "./tasks.js"; | |
| 15 | 17 | import { splitPlan } from "./projects.js"; |
| 16 | 18 | |
| 17 | 19 | /** |
| @@ -189,7 +191,7 @@ | ||
| 189 | 191 | return <pre class={`tx-pre${item.kind === "error" ? " tx-err" : ""}`}>{item.body}</pre>; |
| 190 | 192 | }; |
| 191 | 193 | |
| 192 | -function taskPage(c: AppContext, l: Loaded, t: TaskView, items: Array<{ run: number; item: TranscriptItem }>, truncated: boolean, canSplit = false) { | |
| 194 | +function taskPage(c: AppContext, l: Loaded, t: TaskView, items: Array<{ run: number; item: TranscriptItem }>, truncated: boolean, canSplit = false, attempts: AttemptCompare[] = []) { | |
| 193 | 195 | const base = repoUrl(l.repo); |
| 194 | 196 | const tk = csrfToken(c); |
| 195 | 197 | const me = requireAccount(c); |
| @@ -278,6 +280,7 @@ | ||
| 278 | 280 | ) : t.plan_state === "planning" ? ( |
| 279 | 281 | <div class="flash info">This task plans first: its run reads the code in a read-only sandbox and writes a plan for {t.person_login} to approve, change or split into tasks.</div> |
| 280 | 282 | ) : null} |
| 283 | + {attempts.length ? <Attempts t={t} attempts={attempts} base={base} tk={tk} starter={starter} /> : null} | |
| 281 | 284 | {t.parent_task ? ( |
| 282 | 285 | <div class="flash info"> |
| 283 | 286 | Split from the plan of <a href={`${base}/tasks/${t.parent_task}`}>task #{t.parent_task}</a>. |
| @@ -297,13 +300,15 @@ | ||
| 297 | 300 | ) : null} |
| 298 | 301 | {items.length ? items.map((x, i) => ( |
| 299 | 302 | <> |
| 300 | - {i === 0 || items[i - 1]!.run !== x.run ? <h4 class="tx-run">Run {x.run}{t.runs[x.run - 1]?.message ? " · follow-up" : ""}</h4> : null} | |
| 303 | + {i === 0 || items[i - 1]!.run !== x.run ? <h4 class="tx-run">Run {x.run}{t.runs[x.run - 1]?.attempt ? ` · attempt ${t.runs[x.run - 1]!.attempt}` : ""}{t.runs[x.run - 1]?.message ? " · follow-up" : ""}</h4> : null} | |
| 301 | 304 | <ItemView item={x.item} base={base} /> |
| 302 | 305 | </> |
| 303 | 306 | )) : <div class="empty">Starting…</div>} |
| 304 | 307 | </div> |
| 305 | 308 | </div> |
| 306 | - {starter ? ( | |
| 309 | + {starter && t.attempts > 1 && (t.pick_state === "running" || t.pick_state === "waiting") ? ( | |
| 310 | + <p class="muted small">{t.pick_state === "running" ? `Follow-ups open once you've picked one of the ${t.attempts} attempts.` : "Pick an attempt to continue: follow-ups go to the picked attempt's branch."}</p> | |
| 311 | + ) : starter ? ( | |
| 307 | 312 | <Form action={`${base}/tasks/${t.number}/messages`} t={tk} class="box"> |
| 308 | 313 | <div class="head"> |
| 309 | 314 | <h2>{t.plan_state === "waiting" ? "Ask for changes to the plan" : running ? "Send a follow-up" : "Continue the task"}</h2> |
| @@ -330,7 +335,63 @@ | ||
| 330 | 335 | ); |
| 331 | 336 | } |
| 332 | 337 | |
| 333 | -async function newTaskPage(c: AppContext, l: Loaded, error?: string, v: { goal?: string; runtime?: string; agent?: string; base?: string; command?: string; model?: string; pull?: string; plan_first?: string } = {}) { | |
| 338 | +/** Best of N: the attempts side by side, with Pick. */ | |
| 339 | +const Attempts: FC<{ t: TaskView; attempts: AttemptCompare[]; base: string; tk: string; starter: boolean }> = ({ t, attempts, base, tk, starter }) => { | |
| 340 | + const waiting = t.pick_state === "waiting"; | |
| 341 | + return ( | |
| 342 | + <div class="box" id="attempts"> | |
| 343 | + <div class="head"> | |
| 344 | + <h2>Attempts</h2> | |
| 345 | + {waiting ? <span class="badge amber">Pick one</span> : t.pick_state === "picked" ? <span class="badge green">Attempt {t.picked} picked</span> : t.pick_state === "running" ? <span class="badge blue">Running</span> : null} | |
| 346 | + </div> | |
| 347 | + <div class="body"> | |
| 348 | + <table class="tbl"> | |
| 349 | + <thead> | |
| 350 | + <tr> | |
| 351 | + <th>Attempt</th> | |
| 352 | + <th>Changes</th> | |
| 353 | + <th>Checks</th> | |
| 354 | + <th>Tokens</th> | |
| 355 | + <th></th> | |
| 356 | + </tr> | |
| 357 | + </thead> | |
| 358 | + <tbody> | |
| 359 | + {attempts.map((a) => ( | |
| 360 | + <tr id={`attempt-${a.attempt}`}> | |
| 361 | + <td> | |
| 362 | + <strong>Attempt {a.attempt}</strong> | |
| 363 | + <div class="muted small mono">{a.head_sha && t.pick_state !== "picked" ? <a href={`${base}/compare/${t.base}...${a.branch}`}>{a.branch}</a> : a.branch}</div> | |
| 364 | + </td> | |
| 365 | + <td>{a.head_sha ? <span>{a.files} {a.files === 1 ? "file" : "files"}, <span style="color:var(--green)">+{a.additions}</span> <span style="color:var(--red)">−{a.deletions}</span></span> : <span class="muted">{a.step === "done" ? "No changes" : a.step === "failed" ? `Failed${a.error ? `: ${a.error}` : ""}` : a.step === "stopped" ? "Stopped" : "Running"}</span>}</td> | |
| 366 | + <td>{a.head_sha ? <span class={`badge ${a.checks.state === "success" ? "green" : a.checks.state === "failure" ? "red" : a.checks.state === "pending" ? "blue" : ""}`}>{a.checks.state === "none" ? "No checks" : a.checks.state === "success" ? "Passing" : a.checks.state === "failure" ? `Failing: ${a.checks.failing.join(", ")}` : "Running"}</span> : null}</td> | |
| 367 | + <td class="muted small">{a.tokens ? `${formatTokens(a.tokens)} (${a.model_calls} ${a.model_calls === 1 ? "request" : "requests"})` : "—"}</td> | |
| 368 | + <td> | |
| 369 | + {waiting && starter && a.head_sha && a.step === "done" ? ( | |
| 370 | + <Form action={`${base}/tasks/${t.number}/attempts/${a.attempt}/pick`} t={tk} class="inline" confirm={`Pick attempt ${a.attempt}? Its branch becomes the task's branch and opens a pull request; the other attempts' branches are deleted.`}> | |
| 371 | + <button class="btn sm primary">Pick</button> | |
| 372 | + </Form> | |
| 373 | + ) : t.picked === a.attempt ? ( | |
| 374 | + <span class="badge green">Picked</span> | |
| 375 | + ) : null} | |
| 376 | + </td> | |
| 377 | + </tr> | |
| 378 | + ))} | |
| 379 | + </tbody> | |
| 380 | + </table> | |
| 381 | + {attempts.map((a) => | |
| 382 | + a.summary ? ( | |
| 383 | + <details class="mt"> | |
| 384 | + <summary>What attempt {a.attempt} said last</summary> | |
| 385 | + <Markdown text={a.summary} /> | |
| 386 | + </details> | |
| 387 | + ) : null, | |
| 388 | + )} | |
| 389 | + </div> | |
| 390 | + </div> | |
| 391 | + ); | |
| 392 | +}; | |
| 393 | + | |
| 394 | +async function newTaskPage(c: AppContext, l: Loaded, error?: string, v: { goal?: string; runtime?: string; agent?: string; base?: string; command?: string; model?: string; pull?: string; plan_first?: string; attempts?: string } = {}) { | |
| 334 | 395 | const me = requireAccount(c); |
| 335 | 396 | const base = repoUrl(l.repo); |
| 336 | 397 | const tk = csrfToken(c); |
| @@ -400,6 +461,13 @@ | ||
| 400 | 461 | label={<>Plan first: the agent reads the code and writes a plan in a read-only sandbox; it builds once you approve the plan{settings.planFirst === "required" ? " (this project requires it)" : ""}</>} |
| 401 | 462 | /> |
| 402 | 463 | </div> |
| 464 | + {v.pull ? null : ( | |
| 465 | + <div class="field" style="max-width:320px"> | |
| 466 | + <label for="attempts">Attempts</label> | |
| 467 | + <KSelect id="attempts" name="attempts" block label="Attempts" value={v.attempts ?? "1"} options={Array.from({ length: MAX_ATTEMPTS }, (_, i) => ({ value: String(i + 1), label: i === 0 ? "1 attempt" : `${i + 1} attempts, then pick the best` }))} /> | |
| 468 | + <p class="muted small">Each attempt runs in its own sandbox on its own branch, at the same time, and counts toward the project's limit. Each spends your model key.</p> | |
| 469 | + </div> | |
| 470 | + )} | |
| 403 | 471 | <button class="btn primary" disabled={!agents.length}> |
| 404 | 472 | Start task |
| 405 | 473 | </button> |
| @@ -482,7 +550,7 @@ | ||
| 482 | 550 | |
| 483 | 551 | post(web, "/:owner/:repo/tasks", async (c, f) => { |
| 484 | 552 | const l = await forTasks(c); |
| 485 | - const v = { goal: f.s("goal"), runtime: f.s("runtime") || "claude-code", agent: f.s("agent"), base: f.s("base"), command: f.s("command"), model: f.s("model"), pull: f.s("pull") || undefined, plan_first: f.all("plan_first").length ? "1" : "" }; | |
| 553 | + const v = { goal: f.s("goal"), runtime: f.s("runtime") || "claude-code", agent: f.s("agent"), base: f.s("base"), command: f.s("command"), model: f.s("model"), pull: f.s("pull") || undefined, plan_first: f.all("plan_first").length ? "1" : "", attempts: f.s("attempts") || "1" }; | |
| 486 | 554 | try { |
| 487 | 555 | const { runtime, provider } = parseRuntimeChoice(v.runtime); |
| 488 | 556 | const t = await startTask(c, l, { ...v, runtime, provider, model: provider ? v.model : undefined, plan_first: v.plan_first === "1" }); |
| @@ -504,7 +572,8 @@ | ||
| 504 | 572 | const tail = lines.slice(-1500); |
| 505 | 573 | const items = tail.flatMap((line) => transcriptItem(line).map((item) => ({ run: line.run, item }))); |
| 506 | 574 | const canSplit = t.plan_state === "waiting" && !!(await stub.coordinator(requireAccount(c).id)); |
| 507 | - return taskPage(c, l, t, items, lines.length > tail.length, canSplit); | |
| 575 | + const attempts = t.attempts > 1 ? await compareAttempts(c, l, n) : []; | |
| 576 | + return taskPage(c, l, t, items, lines.length > tail.length, canSplit, attempts); | |
| 508 | 577 | }); |
| 509 | 578 | |
| 510 | 579 | post(web, "/:owner/:repo/tasks/:number/messages", async (c, f) => { |
| @@ -515,6 +584,15 @@ | ||
| 515 | 584 | }); |
| 516 | 585 | }); |
| 517 | 586 | |
| 587 | + post(web, "/:owner/:repo/tasks/:number/attempts/:attempt/pick", async (c) => { | |
| 588 | + const l = await forTasks(c); | |
| 589 | + const n = taskNumber(c.req.param("number")); | |
| 590 | + const k = Number(prm(c, "attempt")); | |
| 591 | + return act(c, `${repoUrl(l.repo)}/tasks/${n}`, async () => { | |
| 592 | + await pickAttempt(c, l, n, k); | |
| 593 | + }, `Attempt ${k} picked.`); | |
| 594 | + }); | |
| 595 | + | |
| 518 | 596 | post(web, "/:owner/:repo/tasks/:number/plan/approve", async (c) => { |
| 519 | 597 | const l = await forTasks(c); |
| 520 | 598 | const n = taskNumber(c.req.param("number")); |
modifiedapps/diffinery/src/agentic/tasks.ts+56−4View file
| @@ -6,7 +6,9 @@ | ||
| 6 | 6 | import { atLeast, type Caller } from "../repo/types.js"; |
| 7 | 7 | import { mayDriveAgent, taskCaller, TaskAccessError } from "./access.js"; |
| 8 | 8 | import { isProvider, ModelKeys, PROVIDERS, type Provider } from "./keys.js"; |
| 9 | -import type { TaskRow, TaskView } from "./ProjectDO.js"; | |
| 9 | +import type { AttemptView, TaskRow, TaskView } from "./ProjectDO.js"; | |
| 10 | +import { MAX_ATTEMPTS } from "./project.js"; | |
| 11 | +import { checksState, type ChecksState } from "./today-repo.js"; | |
| 10 | 12 | import { isRuntime, MAX_COMMAND, MAX_GOAL, OPENROUTER_DEFAULT_MODEL, OPENROUTER_MODEL, RUNTIMES, runtimeLabel, runtimeNeeds, type ModelRoute, type RuntimeId, type TranscriptLine } from "./runtimes.js"; |
| 11 | 13 | |
| 12 | 14 | /** |
| @@ -102,6 +104,8 @@ | ||
| 102 | 104 | pull?: unknown; |
| 103 | 105 | /** Plan first: true or false; left out, the project's default. A project can require it. */ |
| 104 | 106 | plan_first?: unknown; |
| 107 | + /** Best of N: run the first build as 1 to 4 attempts, then pick one. */ | |
| 108 | + attempts?: unknown; | |
| 105 | 109 | } |
| 106 | 110 | |
| 107 | 111 | export async function startTask(c: AppContext, l: Loaded, input: StartInput): Promise<TaskView> { |
| @@ -134,12 +138,15 @@ | ||
| 134 | 138 | } |
| 135 | 139 | if (!(await l.stub.getRef(`refs/heads/${base}`))) throw new HttpError(422, `branch ${base} doesn't exist`); |
| 136 | 140 | if (input.plan_first !== undefined && typeof input.plan_first !== "boolean") throw new HttpError(422, "`plan_first` must be true or false"); |
| 141 | + const attempts = input.attempts === undefined || input.attempts === null || input.attempts === "" ? 1 : Number(input.attempts); | |
| 142 | + if (!Number.isInteger(attempts) || attempts < 1 || attempts > MAX_ATTEMPTS) throw new HttpError(422, `\`attempts\` must be a whole number from 1 to ${MAX_ATTEMPTS}`); | |
| 143 | + if (attempts > 1 && pull) throw new HttpError(422, "a task on a pull request's branch runs one attempt"); | |
| 137 | 144 | const providers = await providersFor(c.env, person, runtime, route.via); |
| 138 | 145 | const project = projectStub(c.env, l.repo.id); |
| 139 | 146 | const { planFirst: policy } = await project.projectSettings(); |
| 140 | 147 | const planFirst = policy === "required" || (input.plan_first === undefined ? policy === "default" : input.plan_first === true); |
| 141 | - const t = await project.create({ repoId: l.repo.id, goal, runtime, command, route, agent: { id: agent.id, login: agent.login }, person: { id: person.id, login: person.login }, base, providers, pull, planFirst }); | |
| 142 | - c.executionCtx.waitUntil(c.get("dir").audit({ actorId: person.id, actorLogin: person.login, action: "task.start", target: `${l.repo.owner_login}/${l.repo.name}#${t.number}`, detail: { agent: agent.login, runtime, ...(route.via ? { provider: route.via, model: route.model } : {}), ...(planFirst ? { plan_first: true } : {}) } })); | |
| 148 | + const t = await project.create({ repoId: l.repo.id, goal, runtime, command, route, agent: { id: agent.id, login: agent.login }, person: { id: person.id, login: person.login }, base, providers, pull, planFirst, attempts }); | |
| 149 | + c.executionCtx.waitUntil(c.get("dir").audit({ actorId: person.id, actorLogin: person.login, action: "task.start", target: `${l.repo.owner_login}/${l.repo.name}#${t.number}`, detail: { agent: agent.login, runtime, ...(route.via ? { provider: route.via, model: route.model } : {}), ...(planFirst ? { plan_first: true } : {}), ...(attempts > 1 ? { attempts } : {}) } })); | |
| 143 | 150 | return t; |
| 144 | 151 | } |
| 145 | 152 | |
| @@ -167,6 +174,49 @@ | ||
| 167 | 174 | return t; |
| 168 | 175 | } |
| 169 | 176 | |
| 177 | +/** Best of N: pick one of a task's attempts, which becomes its branch and pull request. */ | |
| 178 | +export async function pickAttempt(c: AppContext, l: Loaded, number: number, attempt: number): Promise<TaskView> { | |
| 179 | + const { person } = await starter(c, l, number); | |
| 180 | + const t = await projectStub(c.env, l.repo.id).pick(number, attempt, person.login); | |
| 181 | + c.executionCtx.waitUntil(c.get("dir").audit({ actorId: person.id, actorLogin: person.login, action: "task.pick", target: `${l.repo.owner_login}/${l.repo.name}#${number}`, detail: { attempt } })); | |
| 182 | + return t; | |
| 183 | +} | |
| 184 | + | |
| 185 | +/** One attempt as the comparison shows it: what it changed, its checks, what it cost, what it said. */ | |
| 186 | +export interface AttemptCompare extends AttemptView { | |
| 187 | + files: number; | |
| 188 | + additions: number; | |
| 189 | + deletions: number; | |
| 190 | + checks: { state: ChecksState; failing: string[] }; | |
| 191 | +} | |
| 192 | + | |
| 193 | +export async function compareAttempts(c: AppContext, l: Loaded, number: number): Promise<AttemptCompare[]> { | |
| 194 | + const list = await projectStub(c.env, l.repo.id).attempts(number); | |
| 195 | + return Promise.all( | |
| 196 | + list.map(async (a) => { | |
| 197 | + if (!a.head_sha) return { ...a, files: 0, additions: 0, deletions: 0, checks: { state: "none" as ChecksState, failing: [] } }; | |
| 198 | + const [d, checks] = await Promise.all([l.stub.diff(a.start_sha, a.head_sha, { patch: false }).catch(() => null), l.stub.checksAndStatuses(a.head_sha).catch(() => [])]); | |
| 199 | + return { ...a, files: d?.totalFiles ?? 0, additions: d?.additions ?? 0, deletions: d?.deletions ?? 0, checks: checksState(checks) }; | |
| 200 | + }), | |
| 201 | + ); | |
| 202 | +} | |
| 203 | + | |
| 204 | +export const attemptJson = (base: string, repoFull: string, a: AttemptCompare) => ({ | |
| 205 | + attempt: a.attempt, | |
| 206 | + run: a.run, | |
| 207 | + state: a.step, | |
| 208 | + error: a.error, | |
| 209 | + branch: a.branch, | |
| 210 | + head_sha: a.head_sha, | |
| 211 | + changes: { files: a.files, additions: a.additions, deletions: a.deletions }, | |
| 212 | + checks: a.checks.state, | |
| 213 | + failing_checks: a.checks.failing, | |
| 214 | + model_requests: a.model_calls, | |
| 215 | + tokens: a.tokens, | |
| 216 | + summary: a.summary, | |
| 217 | + ...(a.head_sha ? { compare_url: `${base}/${repoFull}/compare/${a.start_sha ?? ""}...${a.branch}` } : {}), | |
| 218 | +}); | |
| 219 | + | |
| 170 | 220 | /** Approve a task's waiting plan: its build run starts. */ |
| 171 | 221 | export async function approvePlan(c: AppContext, l: Loaded, number: number): Promise<TaskView> { |
| 172 | 222 | const { person, task } = await starter(c, l, number); |
| @@ -224,6 +274,8 @@ | ||
| 224 | 274 | plan_first: !!t.plan_first, |
| 225 | 275 | ...(t.plan_first ? { plan_state: t.plan_state, plan: t.plan } : {}), |
| 226 | 276 | ...(t.parent_task ? { parent_task: t.parent_task } : {}), |
| 277 | + attempts: t.attempts ?? 1, | |
| 278 | + ...((t.attempts ?? 1) > 1 ? { pick_state: t.pick_state, picked_attempt: t.picked } : {}), | |
| 227 | 279 | // Security 3b: a task a coordinator started says so, with the post it came from. |
| 228 | 280 | ...(t.coordinator_post ? { started_via: { type: "coordinator", post: t.coordinator_post }, token_budget: t.token_budget, waits_for: t.after_tasks ? (JSON.parse(t.after_tasks) as number[]) : [] } : {}), |
| 229 | 281 | base: t.base, |
| @@ -234,7 +286,7 @@ | ||
| 234 | 286 | html_url: `${base}/${repoFull}/tasks/${t.number}`, |
| 235 | 287 | created_at: t.created_at, |
| 236 | 288 | updated_at: t.updated_at, |
| 237 | - ...("runs" in t ? { runs: t.runs.map((r) => ({ number: r.seq, kind: r.kind, message: r.message, state: r.step, exit_code: r.exit_code, error: r.error, model_requests: r.model_calls ?? 0, tokens: r.tokens ?? 0, started_at: r.started_at, ended_at: r.ended_at })) } : {}), | |
| 289 | + ...("runs" in t ? { runs: t.runs.map((r) => ({ number: r.seq, kind: r.kind, message: r.message, state: r.step, exit_code: r.exit_code, error: r.error, model_requests: r.model_calls ?? 0, tokens: r.tokens ?? 0, ...(r.attempt ? { attempt: r.attempt } : {}), started_at: r.started_at, ended_at: r.ended_at })) } : {}), | |
| 238 | 290 | }; |
| 239 | 291 | } |
| 240 | 292 |
modifiedapps/diffinery/test/e2e/tasks.test.mjs+98−0View file
| @@ -885,6 +885,104 @@ | ||
| 885 | 885 | assert.equal((await asPerson("DELETE", `${P}/coordinator`)).status, 204); |
| 886 | 886 | }); |
| 887 | 887 | |
| 888 | +test("best of N: attempts run side by side on their own branches, then one is picked", async () => { | |
| 889 | + const me = await ok("GET", "/user"); | |
| 890 | + const owner = me.login; | |
| 891 | + const P = `/repos/${owner}/tasks-demo/diffinery/project`; | |
| 892 | + const R = `/repos/${owner}/tasks-demo/diffinery/tasks`; | |
| 893 | + const src = join(work, "src"); | |
| 894 | + // Attempts get the same prompt; each claims a slot (the e2e pool's sandboxes share /tmp) and does different work. | |
| 895 | + writeFileSync( | |
| 896 | + join(src, "best.sh"), | |
| 897 | + `p="$DIFFINERY_TASK_PROMPT" | |
| 898 | +case "$p" in | |
| 899 | + *"New instructions from"*) echo follow > follow.txt; echo FOLLOWED-UP; exit 0 ;; | |
| 900 | +esac | |
| 901 | +tag=$(printf '%s\\n' "$p" | grep -o 'BEST-[a-z0-9]*' | head -1) | |
| 902 | +slot=0 | |
| 903 | +for i in 1 2 3 4; do if mkdir "/tmp/$tag-$i" 2>/dev/null; then slot=$i; break; fi; done | |
| 904 | +case $slot in | |
| 905 | + 1) echo one > one.txt; echo SUMMARY-ONE-FILE ;; | |
| 906 | + 2) echo a > a.txt; echo b > b.txt; echo c > c.txt; echo SUMMARY-THREE-FILES ;; | |
| 907 | + *) echo SLOT-FAIL; exit 3 ;; | |
| 908 | +esac | |
| 909 | +`, | |
| 910 | + ); | |
| 911 | + git(src, ["pull", "-q", "--rebase", "origin", "main"]); | |
| 912 | + git(src, ["add", "best.sh"]); | |
| 913 | + git(src, ["commit", "-q", "-m", "best of N script"]); | |
| 914 | + git(src, ["push", "-q", "origin", "main"]); | |
| 915 | + const tag = `BEST-${randomBytes(6).toString("hex")}`; | |
| 916 | + const start = { goal: `${tag}: try it three ways`, runtime: "custom", command: "sh best.sh", agent: "task-bot" }; | |
| 917 | + | |
| 918 | + for (const bad of [0, 5, 1.5, "x"]) assert.equal((await asPerson("POST", R, { ...start, attempts: bad })).status, 422, String(bad)); | |
| 919 | + const started = await asPerson("POST", R, { ...start, attempts: 3 }); | |
| 920 | + assert.equal(started.status, 201, JSON.stringify(started.json)); | |
| 921 | + const n = started.json.number; | |
| 922 | + assert.equal(started.json.attempts, 3); | |
| 923 | + assert.equal(started.json.pick_state, "running"); | |
| 924 | + assert.deepEqual(started.json.runs.map((r) => r.attempt), [1, 2, 3]); | |
| 925 | + | |
| 926 | + // They run together, each on its own branch; the task waits on you when the last one ends. | |
| 927 | + const done = await until("the attempts", async () => { | |
| 928 | + const t = await ok("GET", `${R}/${n}`); | |
| 929 | + return !["running", "queued"].includes(t.state) ? t : null; | |
| 930 | + }); | |
| 931 | + const text = (await ok("GET", `${R}/${n}/transcript`)).map((l) => l.text).join("\n"); | |
| 932 | + assert.equal(done.state, "ready", text); | |
| 933 | + assert.equal(done.pick_state, "waiting"); | |
| 934 | + assert.equal(done.pull_request, null, "no pull request until a pick"); | |
| 935 | + assert.match(text, new RegExp(`Attempt 2 of 3, on ${done.branch}-a2\\.`)); | |
| 936 | + assert.ok((await ok("GET", P)).board.waiting.some((t) => t.number === n), "attempts to pick from wait on you"); | |
| 937 | + assert.ok((await ok("GET", `${P}/posts`)).some((p) => p.body === `Task #${n}'s 3 attempts are done: 2 of them pushed work to compare. Pick one.`)); | |
| 938 | + | |
| 939 | + const attempts = await ok("GET", `${R}/${n}/attempts`); | |
| 940 | + assert.equal(attempts.length, 3); | |
| 941 | + assert.deepEqual(attempts.map((a) => a.branch), [done.branch, `${done.branch}-a2`, `${done.branch}-a3`]); | |
| 942 | + const failed = attempts.find((a) => a.state === "failed"); | |
| 943 | + const one = attempts.find((a) => a.changes.files === 1); | |
| 944 | + const three = attempts.find((a) => a.changes.files === 3); | |
| 945 | + assert.ok(failed && one && three, JSON.stringify(attempts)); | |
| 946 | + assert.equal(failed.head_sha, null); | |
| 947 | + assert.match(one.summary, /SUMMARY-ONE-FILE/); | |
| 948 | + assert.match(three.summary, /SUMMARY-THREE-FILES/); | |
| 949 | + assert.deepEqual(three.changes, { files: 3, additions: 3, deletions: 0 }); | |
| 950 | + for (const a of [one, three]) assert.equal(git(src, ["ls-remote", "origin", `refs/heads/${a.branch}`]).trim().split(/\s/)[0], a.head_sha); | |
| 951 | + const page = await (await fetch(`${base}/${owner}/tasks-demo/tasks/${n}`)).text(); | |
| 952 | + assert.match(page, /<h2>Attempts<\/h2>/); | |
| 953 | + assert.match(page, new RegExp(`/tasks/${n}/attempts/${three.attempt}/pick`)); | |
| 954 | + | |
| 955 | + // Follow-ups wait for a pick; only the person who started it may pick, and only an attempt with work. | |
| 956 | + assert.equal((await asPerson("POST", `${R}/${n}/messages`, { body: "go on" })).status, 409); | |
| 957 | + assert.equal((await api("POST", `${R}/${n}/attempts/${three.attempt}/pick`)).status, 403); | |
| 958 | + assert.equal((await asPerson("POST", `${R}/${n}/attempts/${failed.attempt}/pick`)).status, 422); | |
| 959 | + assert.equal((await asPerson("POST", `${R}/${n}/attempts/9/pick`)).status, 404); | |
| 960 | + const picked = await asPerson("POST", `${R}/${n}/attempts/${three.attempt}/pick`); | |
| 961 | + assert.equal(picked.status, 202, JSON.stringify(picked.json)); | |
| 962 | + assert.equal(picked.json.picked_attempt, three.attempt); | |
| 963 | + assert.equal(picked.json.pick_state, "picked"); | |
| 964 | + assert.equal(picked.json.branch, three.branch); | |
| 965 | + assert.ok(picked.json.pull_request, "picking opens the pull request"); | |
| 966 | + const pr = await ok("GET", `/repos/${owner}/tasks-demo/pulls/${picked.json.pull_request.number}`); | |
| 967 | + assert.equal(pr.head.ref, three.branch); | |
| 968 | + assert.equal(git(src, ["ls-remote", "origin", `refs/heads/${one.branch}`]).trim(), "", "the other attempt's branch is deleted"); | |
| 969 | + assert.equal((await asPerson("POST", `${R}/${n}/attempts/${one.attempt}/pick`)).status, 409); | |
| 970 | + assert.ok((await ok("GET", `${P}/posts`)).some((p) => p.body === `${owner} picked attempt ${three.attempt} of task #${n}.`)); | |
| 971 | + | |
| 972 | + // Later runs build on the picked attempt. | |
| 973 | + assert.equal((await asPerson("POST", `${R}/${n}/messages`, { body: "Add a follow file" })).status, 201); | |
| 974 | + const next = await until("the follow-up", async () => { | |
| 975 | + const t = await ok("GET", `${R}/${n}`); | |
| 976 | + return t.runs.length === 4 && !["queued", "boot", "install", "agent", "finish"].includes(t.runs[3].state) ? t : null; | |
| 977 | + }); | |
| 978 | + assert.equal(next.runs[3].state, "done"); | |
| 979 | + assert.equal(next.runs[3].attempt, undefined); | |
| 980 | + assert.match((await ok("GET", `${R}/${n}/transcript`)).map((l) => l.text).join("\n"), new RegExp(`Starting a sandbox on ${three.branch} as task-bot\\.`)); | |
| 981 | + const files = (await ok("GET", `/repos/${owner}/tasks-demo/pulls/${pr.number}/files`)).map((f) => f.filename).sort(); | |
| 982 | + assert.deepEqual(files, ["a.txt", "b.txt", "c.txt", "follow.txt"]); | |
| 983 | + for (let i = 1; i <= 4; i++) rmSync(`/tmp/${tag}-${i}`, { recursive: true, force: true }); | |
| 984 | +}); | |
| 985 | + | |
| 888 | 986 | test("losing access to the agent stops the task", async () => { |
| 889 | 987 | const me = await ok("GET", "/user"); |
| 890 | 988 | const owner = me.login; |
modifiedapps/diffinery/test/unit/project.test.ts+10−1View file
| @@ -1,5 +1,5 @@ | ||
| 1 | 1 | import { describe, expect, it } from "vitest"; |
| 2 | -import { boardState, isPlanFirst, lineDiff, projectContext, validMaxRunning, type PullState } from "../../src/agentic/project.js"; | |
| 2 | +import { attemptBranch, boardState, isPlanFirst, lineDiff, projectContext, validMaxRunning, type PullState } from "../../src/agentic/project.js"; | |
| 3 | 3 | |
| 4 | 4 | const open = (p: Partial<{ checks: "success" | "failure" | "pending" | "none"; approvedHead: boolean; autoMerge: boolean; queued: boolean }> = {}): PullState => ({ open: true, pull: { checks: "success", approvedHead: false, autoMerge: false, queued: false, draft: false, ...p } }); |
| 5 | 5 | |
| @@ -19,6 +19,15 @@ | ||
| 19 | 19 | expect(boardState({ state: "ready", pr_number: 4, plan_state: "approved" }, open())).toBe("review"); |
| 20 | 20 | expect(boardState({ state: "failed", pr_number: null, plan_state: "waiting" }, null)).toBe("waiting"); |
| 21 | 21 | }); |
| 22 | + it("puts attempts waiting for a pick on you", () => { | |
| 23 | + expect(boardState({ state: "ready", pr_number: null, pick_state: "waiting" }, null)).toBe("waiting"); | |
| 24 | + expect(boardState({ state: "running", pr_number: null, pick_state: "running" }, null)).toBe("working"); | |
| 25 | + expect(boardState({ state: "ready", pr_number: 4, pick_state: "picked" }, open())).toBe("review"); | |
| 26 | + }); | |
| 27 | + it("names each attempt's branch beside the task's", () => { | |
| 28 | + expect(attemptBranch("agent/7-add-caching", 1)).toBe("agent/7-add-caching"); | |
| 29 | + expect(attemptBranch("agent/7-add-caching", 3)).toBe("agent/7-add-caching-a3"); | |
| 30 | + }); | |
| 22 | 31 | it("knows the plan-first policies", () => { |
| 23 | 32 | for (const v of ["off", "default", "required"]) expect(isPlanFirst(v)).toBe(true); |
| 24 | 33 | for (const v of ["on", "", null, 1, "toString"]) expect(isPlanFirst(v)).toBe(false); |