Say a runner was stopped, not replaced by a new version, unless a deploy did it (#220)
* Say a runner was stopped, not replaced by a new version, unless a deploy did it
7 files changed+15−9
Files (7)
modifiedapps/diffinery/docs/actions/overview.md+1−1View file
| @@ -451,7 +451,7 @@ | ||
| 451 | 451 | | `The job was not started: Hosted runners are paused on this instance by a site administrator: <reason>. Self-hosted runners still work.` | Hosted compute is paused on the whole instance. Use a self-hosted runner, or re-run once it is resumed. See [Kill switches](/docs/admin/site-administration#kill-switches). | |
| 452 | 452 | | `Could not start a runner: ...` | The job's container failed to start with an error that trying again won't fix (passing platform errors are retried first; see [Limits](/docs/reference/limits)). Re-run the run. | |
| 453 | 453 | | `The runner stopped unexpectedly (exit code <n>).` | The job's container exited before reporting a result. Re-run the run; if it repeats, check that the job doesn't exhaust memory or disk. | |
| 454 | -| `This runner is being replaced by a new version. The job keeps running for up to 13 minutes before it is cancelled, but a step the platform's stop signal ends fails.` | The platform is replacing this job's container. Deploys leave running jobs alone, so this is rare. The platform's stop signal reaches the running step as well, which then fails with "The platform stopped this step's processes while replacing the runner with a new version." Re-run the job. | | |
| 454 | +| `The platform is stopping this runner (a rollout replacing it with a new version, or the platform moving it). The job keeps running for up to 13 minutes; ...` | The platform sent the job's container its stop signal. Deploys leave running jobs alone, so this is rare and usually the platform moving the container. The signal reaches the running step as well, which then fails with "The platform stopped this step's processes while stopping the runner." A job that can only read starts again by itself; see Platform updates in [Runners](/docs/actions/runners#limits-and-timeouts). Otherwise re-run the job. | | |
| 455 | 455 | | `Error evaluating 'if': ...` | The job's `if:` expression failed at runtime. See [Expressions](/docs/actions/expressions). | |
| 456 | 456 | | A step can't reach a secret | Secrets are configured in the repository or organization settings. See [Secrets and variables](/docs/actions/secrets-and-variables). | |
| 457 | 457 | | `git push` from a job fails | The job's `GITHUB_TOKEN` needs `contents: write` (new repositories default to read-only tokens), can't change workflow files, and follows branch protection. See [GITHUB_TOKEN permissions](/docs/actions/workflow-permissions). | |
modifiedapps/diffinery/docs/actions/runners.md+1−1View file
| @@ -183,7 +183,7 @@ | ||
| 183 | 183 | | Matrix size | 256 jobs per matrix | The job fails with "A matrix may produce at most 256 jobs." | |
| 184 | 184 | | Log line length | 16,384 characters | Longer lines are cut and end with `…` | |
| 185 | 185 | | Log size | 16 MiB of text per job | The log ends with "The log is cut off here: a job's log is limited to 16 MiB of text." Later lines are dropped; the job keeps running. | |
| 186 | -| Platform updates | A deploy is set up not to replace the container of a running job, so running jobs normally finish on the version they started with. The platform doesn't guarantee it, though. If it replaces a container anyway, its stop signal reaches the running step too. The same goes when the platform stops or loses a job's container without warning (it ends with no exit code, for example when its host fails). Diffinery then starts the job again from the beginning on a new runner, up to twice, if no step but `actions/checkout` had started or if the job can only read: it uses no secrets (other than `GITHUB_TOKEN`), no deployment environment, and its token can't write. What the stopped attempt uploaded (artifacts, caches) is dropped first, as a re-run would. A job that could change something outside the run fails instead once another step has started, because running that step twice could repeat what it did (a deploy, a publish). Re-run it from the run page | The log shows "This runner is being replaced by a new version" and, on the step, "The platform stopped this step's processes while replacing the runner with a new version." A restarted job's log starts over with "Diffinery started the job again from the beginning on a new runner (restart N of 2)". A failed one says "The platform replaced this job's runner with a new version while step … ran" or "The platform stopped this job's container while step … ran" (or, after the third replacement, "… 3 times") | | |
| 186 | +| Platform updates | A deploy is set up not to replace the container of a running job, so running jobs normally finish on the version they started with. The platform doesn't guarantee it, though. If it replaces a container anyway, its stop signal reaches the running step too. The same goes when the platform stops or loses a job's container without warning (it ends with no exit code, for example when its host fails). Diffinery then starts the job again from the beginning on a new runner, up to twice, if no step but `actions/checkout` had started or if the job can only read: it uses no secrets (other than `GITHUB_TOKEN`), no deployment environment, and its token can't write. What the stopped attempt uploaded (artifacts, caches) is dropped first, as a re-run would. A job that could change something outside the run fails instead once another step has started, because running that step twice could repeat what it did (a deploy, a publish). Re-run it from the run page | The log shows "The platform is stopping this runner" and, on the step, "The platform stopped this step's processes while stopping the runner." A restarted job's log starts over with "Diffinery started the job again from the beginning on a new runner (restart N of 2)". A failed one says "The platform replaced this job's runner with a new version while step … ran" when a deploy replaced it (the runner was on an older image), or "The platform stopped this job's container while step … ran" when it ran the deployed image (or, after the third replacement, "… 3 times") | | |
| 187 | 187 | |
| 188 | 188 | If the container stops unexpectedly (for example it runs out of memory), the job fails with `The runner stopped unexpectedly (exit code N).` A container that stops before its runner fetched the job is replaced instead, since no step ran. |
| 189 | 189 |
modifiedapps/diffinery/src/repo/actions.ts+6−1View file
| @@ -2192,7 +2192,12 @@ | ||
| 2192 | 2192 | |
| 2193 | 2193 | async runnerComplete(jobId: string, token: string, completion: JobCompletion): Promise<void> { |
| 2194 | 2194 | const job = await this.authJob(jobId, token); |
| 2195 | - if (completion.replaced && completion.conclusion !== "success" && !job.labels) return this.runnerReplaced(job); | |
| 2195 | + if (completion.replaced && completion.conclusion !== "success" && !job.labels) { | |
| 2196 | + // The runner can't tell a rollout's SIGTERM from the platform's own; its image can: on the | |
| 2197 | + // deployed one, no deploy replaced it (the platform stopped or moved its container). | |
| 2198 | + const image = job.runner_info ? (JSON.parse(job.runner_info) as JobRunnerInfo).image : undefined; | |
| 2199 | + return this.runnerReplaced(job, image && this.env.RUNNER_IMAGE && image === this.env.RUNNER_IMAGE ? "stopped" : "replaced"); | |
| 2200 | + } | |
| 2196 | 2201 | await this.completeJob(jobId, completion); |
| 2197 | 2202 | } |
| 2198 | 2203 |
modifiedapps/diffinery/test/unit/actions-runtime.test.ts+2−1View file
| @@ -1020,7 +1020,8 @@ | ||
| 1020 | 1020 | await actions.runnerComplete("r1", started.at(-1)!.token, replaced); |
| 1021 | 1021 | } |
| 1022 | 1022 | expect(job("r1")).toMatchObject({ status: "completed", conclusion: "failure", restarts: MAX_RUNNER_RESTARTS }); |
| 1023 | - expect(JSON.parse(job("r1")!.failure as string).message).toMatch(/replaced this job's runner with a new version after its steps had started, 3 times\. Re-run the job\./); | |
| 1023 | + // (On the deployed image no deploy replaced it, so the message says the platform stopped it.) | |
| 1024 | + expect(JSON.parse(job("r1")!.failure as string).message).toBe("The platform stopped this job's container after its steps had started, 3 times. Re-run the job."); | |
| 1024 | 1025 | // One that names a secret, deploys to an environment, can get an OIDC token, or reads every secret fails as before, naming the step that was stopped. |
| 1025 | 1026 | for (const id of ["r2", "r3", "r4", "r5"]) { |
| 1026 | 1027 | await launch(id); |
modifiedcontainers/runner/src/agent.ts+1−1View file
| @@ -53,7 +53,7 @@ | ||
| 53 | 53 | if (!o.drainOnSigtermMs || draining) return onSignal("SIGTERM"); |
| 54 | 54 | const minutes = Math.round(o.drainOnSigtermMs / 60_000); |
| 55 | 55 | runner.replacing = true; |
| 56 | - runner.logger.line(-1, `##[warning]This runner is being replaced by a new version. The job keeps running for up to ${minutes} minutes; if the platform stops a step or the time runs out, Diffinery starts the job again on a new runner.`); | |
| 56 | + runner.logger.line(-1, `##[warning]The platform is stopping this runner (a rollout replacing it with a new version, or the platform moving it). The job keeps running for up to ${minutes} minutes; if the platform stops a step or the time runs out, the job ends, and Diffinery starts it again on a new runner when doing so can't repeat anything outside the run.`); | |
| 57 | 57 | draining = setTimeout(() => onSignal("SIGTERM"), o.drainOnSigtermMs); |
| 58 | 58 | draining.unref(); |
| 59 | 59 | }; |
modifiedcontainers/runner/src/job.ts+2−2View file
| @@ -653,9 +653,9 @@ | ||
| 653 | 653 | }); |
| 654 | 654 | await this.applyFileCommands(files, o.io); |
| 655 | 655 | if (res.signal && this.replacing && !this.stopping) { |
| 656 | - // A rollout's SIGTERM reaches every process in the container, not only the agent. The job | |
| 656 | + // The platform's SIGTERM (a rollout, or a move) reaches every process in the container, not only the agent. The job | |
| 657 | 657 | // can't succeed now, so stop it and report at once: Diffinery starts it again on a new runner. |
| 658 | - const msg = "The platform stopped this step's processes while replacing the runner with a new version."; | |
| 658 | + const msg = "The platform stopped this step's processes while stopping the runner."; | |
| 659 | 659 | this.logger.line(o.io.logIndex, `##[error]${msg}`); |
| 660 | 660 | this.addAnnotation(o.io.logIndex, { level: "error", message: msg }); |
| 661 | 661 | this.cancel("cancelled"); |
modifiedcontainers/runner/test/job.test.ts+2−2View file
| @@ -819,7 +819,7 @@ | ||
| 819 | 819 | expect(await drain.exited).toBe(0); |
| 820 | 820 | expect(api.completion).toMatchObject({ conclusion: "success" }); |
| 821 | 821 | expect(api.completion).not.toHaveProperty("replaced"); |
| 822 | - expect(api.logText()).toContain("being replaced by a new version"); | |
| 822 | + expect(api.logText()).toContain("The platform is stopping this runner"); | |
| 823 | 823 | expect(api.logText(1)).toContain("after"); |
| 824 | 824 | |
| 825 | 825 | // A second SIGTERM: kill the running step's process group and report cancelled. |
| @@ -862,6 +862,6 @@ | ||
| 862 | 862 | expect(api.completion).toMatchObject({ conclusion: "cancelled", replaced: true }); |
| 863 | 863 | expect(api.lastReport(0)?.conclusion).toBe("cancelled"); |
| 864 | 864 | expect(api.lastReport(1)?.conclusion).toBe("cancelled"); |
| 865 | - expect(api.logText()).toContain("The platform stopped this step's processes while replacing the runner"); | |
| 865 | + expect(api.logText()).toContain("The platform stopped this step's processes while stopping the runner"); | |
| 866 | 866 | }, 60_000); |
| 867 | 867 | }); |