Docker on hosted runners: a regression test for the daemon start lock, and no waiting on it for good (#226)
* Docker on hosted runners: a test for the daemon start lock, and a docker command never waits on it for good
- CI / Typecheck and bundlesuccess
- CI / Unit tests (1/2)success
- CI / Unit tests (2/2)success
- Deploy / Deploy to Cloudflaresuccess
- Deploy / Main's current tipsuccess
4 files changed+105−3
Files (4)
modifiedapps/diffinery/docs/actions/docker.md+1−1View file
| @@ -14,7 +14,7 @@ | ||
| 14 | 14 | |
| 15 | 15 | On hosted runners: |
| 16 | 16 | |
| 17 | -- The `docker` command starts the daemon on first use and prints `Starting the Docker daemon`. Jobs that never use Docker don't start it. A job that will use Docker (a job `container:`, `services:`, a `docker://` action, or a step whose `uses:` or `run:` names `docker`) gets the daemon started in the background as the job begins, so it is usually ready by the time checkout and setup are done. | |
| 17 | +- The `docker` command starts the daemon on first use and prints `Starting the Docker daemon`. Jobs that never use Docker don't start it. A docker command that has waited over 2 minutes for another one to start the daemon fails the step with `The Docker daemon did not start:` instead of hanging. | |
| 18 | 18 | - Docker Hub images are pulled through the registry mirror `https://mirror.gcr.io`, which avoids Docker Hub's anonymous pull limits for official images. Images the mirror doesn't have come from Docker Hub directly. |
| 19 | 19 | - Containers use the DNS servers `1.1.1.1` and `8.8.8.8`. |
| 20 | 20 | - Bridge networks, service names, port publishing and image builds all work, as on GitHub's hosted runners. |
modifiedapps/diffinery/docs/actions/runners.md+1−1View file
| @@ -155,7 +155,7 @@ | ||
| 155 | 155 | |
| 156 | 156 | Docker works on hosted runners the same way as on GitHub's: job containers, service containers, Docker container actions, and `docker build`, `docker run` and `docker compose` in `run:` steps. Details: |
| 157 | 157 | |
| 158 | -- **Started on first use.** The `docker` command is a wrapper that starts the Docker daemon the first time anything in the job needs it (a `docker` command in a step, a job `container:`, `services:`, or a Docker action). The first use prints `Starting the Docker daemon` and takes a few seconds; jobs that never touch Docker don't pay for it. A job that will use Docker (a job `container:`, `services:`, a `docker://` action, or a step whose `uses:` or `run:` names `docker`) gets the daemon started in the background as the job begins, so those seconds overlap checkout and setup. If the daemon can't start, the step fails with `The Docker daemon did not start:` followed by the end of the daemon's log. | |
| 158 | +- **Started on first use.** The `docker` command is a wrapper that starts the Docker daemon the first time anything in the job needs it (a `docker` command in a step, a job `container:`, `services:`, or a Docker action). The first use prints `Starting the Docker daemon` and takes a few seconds; jobs that never touch Docker don't pay for it. Docker commands that arrive while another one is starting the daemon wait for it. If the daemon can't start, or another command has been starting it for over 2 minutes, the step fails with `The Docker daemon did not start:` followed by the end of the daemon's log, rather than waiting until the job's timeout. | |
| 159 | 159 | - **Docker Hub through `mirror.gcr.io`.** The daemon uses `https://mirror.gcr.io` as its registry mirror, so pulls of Docker Hub images (`redis:7-alpine`, `postgres:16`) usually come from Google's public mirror and avoid Docker Hub's anonymous rate limits. Images the mirror doesn't have are pulled from Docker Hub as usual. |
| 160 | 160 | - **DNS.** The daemon gives containers the DNS servers `1.1.1.1` and `8.8.8.8`. |
| 161 | 161 | - **Pushing to the Diffinery registry.** The daemon and BuildKit builders reach `diffinery.com` through a local relay that splits image layer uploads into chunks, so layers larger than Cloudflare's 100 MB request limit push from hosted workflows. See [Container registry](/docs/packages/container-registry). |
modifiedcontainers/runner/image/docker+7−1View file
| @@ -45,7 +45,13 @@ | ||
| 45 | 45 | |
| 46 | 46 | if [ ! -e "$ready" ] && [ -z "${DOCKER_HOST:-}" ] && [ "$(id -u)" = 0 ]; then |
| 47 | 47 | exec 9>/run/diffinery-dockerd.lock |
| 48 | - flock 9 | |
| 48 | + # Starting the daemon takes at most about a minute and a half (the relay, then dockerd), so a | |
| 49 | + # command that waits longer for another one to start it gives up rather than hang the step. | |
| 50 | + if ! flock -w 150 9 && [ ! -e "$ready" ]; then | |
| 51 | + echo "The Docker daemon did not start: another docker command has been starting it for over 2 minutes." >&2 | |
| 52 | + tail -n 20 /var/log/dockerd.log >&2 2>/dev/null | |
| 53 | + exit 1 | |
| 54 | + fi | |
| 49 | 55 | if [ ! -e "$ready" ]; then |
| 50 | 56 | if ! "$real" version >/dev/null 2>&1; then |
| 51 | 57 | echo "Starting the Docker daemon" >&2 |
addedcontainers/runner/test/docker-wrapper.test.ts+96−0View file
| @@ -1,0 +1,96 @@ | ||
| 1 | +import { spawnSync } from "node:child_process"; | |
| 2 | +import { chmodSync, existsSync, readFileSync, rmSync, writeFileSync } from "node:fs"; | |
| 3 | +import path from "node:path"; | |
| 4 | +import { fileURLToPath } from "node:url"; | |
| 5 | +import { afterEach, beforeEach, describe, expect, it } from "vitest"; | |
| 6 | +import { apiBaseFor, runAgentJob } from "../src/agent.js"; | |
| 7 | +import { makeSpec, startFakeApi, tmp, type FakeApi } from "./helpers.js"; | |
| 8 | + | |
| 9 | +// The hosted runner image's docker wrapper (image/docker), run as it is apart from its fixed paths, | |
| 10 | +// against a stand-in daemon that takes a moment to come up and then keeps running, like dockerd. | |
| 11 | +const WRAPPER = path.join(path.dirname(fileURLToPath(import.meta.url)), "../image/docker"); | |
| 12 | +const tools = ["flock", "setsid"].every((t) => spawnSync("sh", ["-c", `command -v ${t}`]).status === 0); | |
| 13 | + | |
| 14 | +let api: FakeApi; | |
| 15 | +let dir: string; | |
| 16 | + | |
| 17 | +/** Writes the wrapper to `dir/bin/docker`, with its paths moved into `dir`, and the stand-ins next to it. */ | |
| 18 | +function install(): string { | |
| 19 | + const bin = path.join(dir, "bin"); | |
| 20 | + const sh = (name: string, body: string) => { | |
| 21 | + writeFileSync(path.join(bin, name), `#!/bin/bash\n${body}\n`); | |
| 22 | + chmodSync(path.join(bin, name), 0o755); | |
| 23 | + }; | |
| 24 | + spawnSync("mkdir", ["-p", bin, path.join(dir, "log")]); | |
| 25 | + let script = readFileSync(WRAPPER, "utf8"); | |
| 26 | + const moves: Array<[string, string]> = [ | |
| 27 | + ["real=/usr/bin/docker", `real=${bin}/docker-real`], | |
| 28 | + ["ready=/run/diffinery-dockerd.ready", `ready=${dir}/dockerd.ready`], | |
| 29 | + ["state=/run/diffinery\n", `state=${dir}/state\n`], | |
| 30 | + ["/run/diffinery-dockerd.lock", `${dir}/dockerd.lock`], | |
| 31 | + ["/var/log/", `${dir}/log/`], | |
| 32 | + ["sys_path=/usr/local/sbin:", `sys_path=${bin}:/usr/local/sbin:`], | |
| 33 | + ['"$(id -u)" = 0', "0 = 0"], | |
| 34 | + ]; | |
| 35 | + for (const [from, to] of moves) { | |
| 36 | + expect(script, `the wrapper no longer has ${JSON.stringify(from)}`).toContain(from); | |
| 37 | + script = script.split(from).join(to); | |
| 38 | + } | |
| 39 | + writeFileSync(path.join(bin, "docker"), script); | |
| 40 | + chmodSync(path.join(bin, "docker"), 0o755); | |
| 41 | + // The daemon answers a second after it starts and keeps running (holding whatever it inherited). | |
| 42 | + sh("dockerd", `echo $$ >> ${dir}/dockerd.pids\nsleep 1\ntouch ${dir}/dockerd.up\nexec sleep 60`); | |
| 43 | + sh("docker-real", `[ -e ${dir}/dockerd.up ] || exit 1\necho "docker $*"`); | |
| 44 | + return bin; | |
| 45 | +} | |
| 46 | + | |
| 47 | +beforeEach(async () => { | |
| 48 | + api = await startFakeApi(); | |
| 49 | + dir = tmp("docker-wrapper-"); | |
| 50 | +}); | |
| 51 | +afterEach(async () => { | |
| 52 | + await api.close(); | |
| 53 | + // Stand-in daemons started with setsid outlive the job, like the real one. | |
| 54 | + const pids = path.join(dir, "dockerd.pids"); | |
| 55 | + if (existsSync(pids)) spawnSync("kill", readFileSync(pids, "utf8").split("\n").filter(Boolean)); | |
| 56 | + rmSync(dir, { recursive: true, force: true }); | |
| 57 | +}); | |
| 58 | + | |
| 59 | +describe.skipIf(!tools)("docker wrapper", () => { | |
| 60 | + it("lets docker commands that arrive while the daemon is starting through, with the head start on", async () => { | |
| 61 | + const bin = install(); | |
| 62 | + // The head start runs `docker version` as the job begins; the step's commands arrive while it is | |
| 63 | + // still starting the daemon, and must not wait on its lock for good. | |
| 64 | + const spec = makeSpec(api, [{ run: "docker info & docker ps & wait\ndocker version" }], { timeoutMinutes: 1 }); | |
| 65 | + const started = Date.now(); | |
| 66 | + const done = await runAgentJob(spec, { | |
| 67 | + apiBase: apiBaseFor(spec.apiUrl, spec.jobId), | |
| 68 | + workRoot: path.join(dir, "work"), | |
| 69 | + toolCache: path.join(dir, "tool"), | |
| 70 | + logIntervalMs: 50, | |
| 71 | + echoLogs: false, | |
| 72 | + prewarmDocker: true, | |
| 73 | + baseEnv: { ...process.env, PATH: `${bin}:${process.env.PATH}` }, | |
| 74 | + }); | |
| 75 | + expect(done.conclusion, api.logText()).toBe("success"); | |
| 76 | + expect(Date.now() - started).toBeLessThan(15_000); | |
| 77 | + expect(api.logText()).toContain("docker info"); | |
| 78 | + expect(api.logText()).toContain("docker ps"); | |
| 79 | + expect(api.logText()).toContain("docker version"); | |
| 80 | + expect(readFileSync(path.join(dir, "log", "dockerd.log"), "utf8")).toBe(""); | |
| 81 | + }, 30_000); | |
| 82 | + | |
| 83 | + it("gives up with a message, not a hang, when the daemon never comes up", () => { | |
| 84 | + const bin = install(); | |
| 85 | + writeFileSync(path.join(bin, "dockerd"), `#!/bin/bash\necho $$ >> ${dir}/dockerd.pids\necho "failed to start daemon: no cgroups" >&2\nexec sleep 60\n`); | |
| 86 | + // Shorten the waits for the test: the start loop and the lock wait. | |
| 87 | + const script = readFileSync(path.join(bin, "docker"), "utf8").replace("seq 1 240", "seq 1 4").replace("flock -w 150 9", "flock -w 2 9"); | |
| 88 | + writeFileSync(path.join(bin, "docker"), script); | |
| 89 | + const env = { ...process.env, PATH: `${bin}:${process.env.PATH}` }; | |
| 90 | + const r = spawnSync("bash", ["-c", "docker version & sleep 0.2; docker info; echo \"second exited $?\"; wait"], { env, encoding: "utf8", timeout: 20_000 }); | |
| 91 | + expect(r.error).toBeUndefined(); | |
| 92 | + expect(r.stdout).toContain("second exited 1"); | |
| 93 | + expect(r.stderr).toContain("The Docker daemon did not start"); | |
| 94 | + expect(r.stderr).toContain("failed to start daemon: no cgroups"); | |
| 95 | + }, 30_000); | |
| 96 | +}); |