Diffinery

codician/diffinerypublic

Open-source, agent-first GitHub and GitHub Actions alternative that runs on your own Cloudflare account: git hosting, pull requests, code review and drop-in GitHub Actions YAML on Workers, Durable Objects, R2 and Containers.

Runner: start the registry relay and the Docker daemon together (#238)

* Runner: start the registry relay and the Docker daemon together
* Runner: a stopped dockerd nobody has reaped yet counts as stopped
test-bot authored 1 parent 920d91bcommit 99277472a820eae6d86e68d79c0502f41f00ec95Browse files
3 files changed+115−17
modifiedapps/diffinery/docs/actions/runners.md+1−1View file
@@ -157,7 +157,7 @@
157157
158158 Docker works on hosted runners the same way as on GitHub's: job containers, service containers, Docker container actions, and `docker build`, `docker run` and `docker compose` in `run:` steps. Details:
159159
160-- **Started on first use.** The `docker` command is a wrapper that starts the Docker daemon the first time anything in the job needs it (a `docker` command in a step, a job `container:`, `services:`, or a Docker action). The first use prints `Starting the Docker daemon` and takes a few seconds; jobs that never touch Docker don't pay for it. A job that will use Docker (a job `container:`, `services:`, a `docker://` action, or a step whose `uses:` or `run:` names `docker`) gets the daemon started in the background as the job begins, so those seconds overlap checkout and setup. Docker commands that arrive while the daemon is starting wait for it. If the daemon can't start, or another command has been starting it for over 2 minutes, the step fails with `The Docker daemon did not start:` followed by the end of the daemon's log, rather than waiting until the job's timeout.
160+- **Started on first use.** The `docker` command is a wrapper that starts the Docker daemon the first time anything in the job needs it (a `docker` command in a step, a job `container:`, `services:`, or a Docker action). The first use prints `Starting the Docker daemon` and takes about 1 to 2 seconds (the daemon and the registry relay below start together); jobs that never touch Docker don't pay for it. A job that will use Docker (a job `container:`, `services:`, a `docker://` action, or a step whose `uses:` or `run:` names `docker`) gets the daemon started in the background as the job begins, so those seconds overlap checkout and setup. Docker commands that arrive while the daemon is starting wait for it. If the daemon can't start, or another command has been starting it for over 2 minutes, the step fails with `The Docker daemon did not start:` followed by the end of the daemon's log, rather than waiting until the job's timeout.
161161 - **Docker Hub through `mirror.gcr.io`.** The daemon uses `https://mirror.gcr.io` as its registry mirror, so pulls of Docker Hub images (`redis:7-alpine`, `postgres:16`) usually come from Google's public mirror and avoid Docker Hub's anonymous rate limits. Images the mirror doesn't have are pulled from Docker Hub as usual.
162162 - **DNS.** The daemon gives containers the DNS servers `1.1.1.1` and `8.8.8.8`.
163163 - **Pushing to the Diffinery registry.** The daemon and BuildKit builders reach `diffinery.com` through a local relay that splits image layer uploads into chunks, so layers larger than Cloudflare's 100 MB request limit push from hosted workflows. See [Container registry](/docs/packages/container-registry).
modifiedcontainers/runner/image/docker+43−10View file
@@ -21,6 +21,8 @@
2121 echo "${url%%/*}"
2222 }
2323
24+# Starts the relay without waiting for it: dockerd starts alongside it, since the daemon only
25+# reaches the relay once it pulls or pushes. relay_up then waits for it.
2426 start_relay() {
2527 local host=$1
2628 mkdir -p "$state" "/etc/docker/certs.d/$host" || return 1
@@ -33,20 +35,41 @@
3335 env -i PATH="$sys_path" ${NODE_EXTRA_CA_CERTS:+"NODE_EXTRA_CA_CERTS=$NODE_EXTRA_CA_CERTS"} DIFFINERY_RELAY_HOST="$host" DIFFINERY_RELAY_UPSTREAM="https://$host" DIFFINERY_RELAY_PORT="$relay_port" \
3436 DIFFINERY_RELAY_CERT="$state/relay.crt" DIFFINERY_RELAY_KEY="$state/relay.key" \
3537 setsid -f node /opt/diffinery/agent.mjs --registry-relay >/var/log/diffinery-relay.log 2>&1 </dev/null 9>&-
36- for _ in $(seq 1 80); do
38+}
39+
40+# Waits up to 20 s for the relay to listen.
41+relay_up() {
42+ for _ in $(seq 1 400); do
3743 if (exec 3<>"/dev/tcp/127.0.0.1/$relay_port") 2>/dev/null; then
38- echo "$host" >"$state/relay.host"
44+ echo "$1" >"$state/relay.host"
3945 return 0
4046 fi
41- sleep 0.25
47+ sleep 0.05
4248 done
4349 return 1
4450 }
4551
52+# Whether process $1 is still running (an exited process nobody has reaped yet is not).
53+running() {
54+ local stat
55+ stat=$(cat "/proc/$1/stat" 2>/dev/null) || return 1
56+ stat=${stat##*) }
57+ [ "${stat%% *}" != Z ]
58+}
59+
60+# Starts dockerd in its own session, noting its pid. The daemons must not inherit the lock (fd 9):
61+# they would hold it for good, and a second docker command that arrives while this one is still
62+# starting the daemon would wait forever.
63+start_dockerd() {
64+ mkdir -p "$state"
65+ env -i PATH="$sys_path" "$@" setsid -f bash -c 'echo $$ >"$1"; exec dockerd' _ "$state/dockerd.pid" >/var/log/dockerd.log 2>&1 </dev/null 9>&-
66+}
67+
4668 if [ ! -e "$ready" ] && [ -z "${DOCKER_HOST:-}" ] && [ "$(id -u)" = 0 ]; then
4769 exec 9>/run/diffinery-dockerd.lock
48- # Starting the daemon takes at most about a minute and a half (the relay, then dockerd), so a
49- # command that waits longer for another one to start it gives up rather than hang the step.
70+ # Starting the daemon takes at most about a minute and a half (the relay and dockerd together,
71+ # and dockerd once more if the relay fails), so a command that waits longer for another one to
72+ # start it gives up rather than hang the step.
5073 if ! flock -w 150 9 && [ ! -e "$ready" ]; then
5174 echo "The Docker daemon did not start: another docker command has been starting it for over 2 minutes." >&2
5275 tail -n 20 /var/log/dockerd.log >&2 2>/dev/null
@@ -56,16 +79,26 @@
5679 if ! "$real" version >/dev/null 2>&1; then
5780 echo "Starting the Docker daemon" >&2
5881 proxy=()
82+ relay=""
5983 if host=$(registry_host) && start_relay "$host"; then
84+ relay=$host
6085 proxy=(HTTPS_PROXY="http://127.0.0.1:$relay_port")
61- elif [ -n "${host:-}" ]; then
86+ fi
87+ start_dockerd "${proxy[@]}"
88+ if [ -n "${host:-}" ] && { [ -z "$relay" ] || ! relay_up "$relay"; }; then
6289 echo "The registry relay did not start, so image layers over 100 MB can't be pushed to $host:" >&2
6390 tail -n 5 /var/log/diffinery-relay.log >&2 2>/dev/null
91+ if [ -n "$relay" ]; then
92+ # The daemon was started with the relay as its proxy: start it again without one.
93+ pid=$(cat "$state/dockerd.pid" 2>/dev/null)
94+ if [ -n "$pid" ] && kill "$pid" 2>/dev/null; then
95+ for _ in $(seq 1 300); do running "$pid" || break; sleep 0.05; done
96+ kill -9 "$pid" 2>/dev/null
97+ fi
98+ start_dockerd
99+ fi
64100 fi
65- # The daemons must not inherit the lock (fd 9): they would hold it for good, and a second
66- # docker command that arrives while this one is still starting the daemon would wait forever.
67- env -i PATH="$sys_path" "${proxy[@]}" setsid -f dockerd >/var/log/dockerd.log 2>&1 </dev/null 9>&-
68- for _ in $(seq 1 240); do "$real" version >/dev/null 2>&1 && break; sleep 0.25; done
101+ for _ in $(seq 1 600); do "$real" version >/dev/null 2>&1 && break; sleep 0.1; done
69102 "$real" version >/dev/null 2>&1 || { echo "The Docker daemon did not start:" >&2; tail -n 20 /var/log/dockerd.log >&2; exit 1; }
70103 fi
71104 touch "$ready"
modifiedcontainers/runner/test/docker-wrapper.test.ts+71−6View file
@@ -1,5 +1,6 @@
11 import { spawnSync } from "node:child_process";
22 import { chmodSync, existsSync, readFileSync, rmSync, writeFileSync } from "node:fs";
3+import { createServer } from "node:net";
34 import path from "node:path";
45 import { fileURLToPath } from "node:url";
56 import { afterEach, beforeEach, describe, expect, it } from "vitest";
@@ -9,13 +10,18 @@
910 // The hosted runner image's docker wrapper (image/docker), run as it is apart from its fixed paths,
1011 // against a stand-in daemon that takes a moment to come up and then keeps running, like dockerd.
1112 const WRAPPER = path.join(path.dirname(fileURLToPath(import.meta.url)), "../image/docker");
12-const tools = ["flock", "setsid"].every((t) => spawnSync("sh", ["-c", `command -v ${t}`]).status === 0);
13+const tools = ["flock", "setsid", "openssl"].every((t) => spawnSync("sh", ["-c", `command -v ${t}`]).status === 0);
1314
1415 let api: FakeApi;
1516 let dir: string;
17+let relayPort: number;
1618
17-/** Writes the wrapper to `dir/bin/docker`, with its paths moved into `dir`, and the stand-ins next to it. */
18-function install(): string {
19+/**
20+ * Writes the wrapper to `dir/bin/docker`, with its paths moved into `dir`, and the stand-ins next to it:
21+ * a daemon that answers `daemonMs` after it starts, and a registry relay that listens `relayMs` after it
22+ * starts (or never, when null).
23+ */
24+function install({ daemonMs = 1000, relayMs = 1000 as number | null } = {}): string {
1925 const bin = path.join(dir, "bin");
2026 const sh = (name: string, body: string) => {
2127 writeFileSync(path.join(bin, name), `#!/bin/bash\n${body}\n`);
@@ -30,6 +36,9 @@
3036 ["/run/diffinery-dockerd.lock", `${dir}/dockerd.lock`],
3137 ["/var/log/", `${dir}/log/`],
3238 ["sys_path=/usr/local/sbin:", `sys_path=${bin}:/usr/local/sbin:`],
39+ ["relay_port=3129", `relay_port=${relayPort}`],
40+ ["/etc/docker/certs.d/", `${dir}/certs.d/`],
41+ ["/opt/diffinery/agent.mjs", `${dir}/relay.mjs`],
3342 ['"$(id -u)" = 0', "0 = 0"],
3443 ];
3544 for (const [from, to] of moves) {
@@ -38,15 +47,38 @@
3847 }
3948 writeFileSync(path.join(bin, "docker"), script);
4049 chmodSync(path.join(bin, "docker"), 0o755);
41- // The daemon answers a second after it starts and keeps running (holding whatever it inherited).
42- sh("dockerd", `echo $$ >> ${dir}/dockerd.pids\nsleep 1\ntouch ${dir}/dockerd.up\nexec sleep 60`);
50+ // The daemon answers after a moment and keeps running (holding whatever it inherited). It notes
51+ // the proxy it was started with.
52+ sh("dockerd", `echo $$ >> ${dir}/dockerd.pids\necho "\${HTTPS_PROXY:-none}" >> ${dir}/dockerd.proxy\nsleep ${daemonMs / 1000}\ntouch ${dir}/dockerd.up\nexec sleep 60`);
4353 sh("docker-real", `[ -e ${dir}/dockerd.up ] || exit 1\necho "docker $*"`);
54+ sh("node", `exec ${process.execPath} "$@"`);
55+ writeFileSync(
56+ path.join(dir, "relay.mjs"),
57+ relayMs === null
58+ ? `process.exit(1);\n`
59+ : `import { appendFileSync } from "node:fs";\nimport { createServer } from "node:net";\nappendFileSync(${JSON.stringify(`${dir}/dockerd.pids`)}, process.pid + "\\n");\nsetTimeout(() => createServer((s) => s.end()).listen(Number(process.env.DIFFINERY_RELAY_PORT), "127.0.0.1"), ${relayMs});\n`,
60+ );
4461 return bin;
4562 }
4663
64+/** Whether process `pid` is running: it exists and isn't a zombie. */
65+function running(pid: string): boolean {
66+ try {
67+ return !/^\S+ \(.*\) Z /.test(readFileSync(`/proc/${pid}/stat`, "utf8"));
68+ } catch {
69+ return false;
70+ }
71+}
72+
4773 beforeEach(async () => {
4874 api = await startFakeApi();
4975 dir = tmp("docker-wrapper-");
76+ relayPort = await new Promise<number>((resolve) => {
77+ const srv = createServer().listen(0, "127.0.0.1", () => {
78+ const port = (srv.address() as { port: number }).port;
79+ srv.close(() => resolve(port));
80+ });
81+ });
5082 });
5183 afterEach(async () => {
5284 await api.close();
@@ -84,7 +116,7 @@
84116 const bin = install();
85117 writeFileSync(path.join(bin, "dockerd"), `#!/bin/bash\necho $$ >> ${dir}/dockerd.pids\necho "failed to start daemon: no cgroups" >&2\nexec sleep 60\n`);
86118 // Shorten the waits for the test: the start loop and the lock wait.
87- const script = readFileSync(path.join(bin, "docker"), "utf8").replace("seq 1 240", "seq 1 4").replace("flock -w 150 9", "flock -w 2 9");
119+ const script = readFileSync(path.join(bin, "docker"), "utf8").replace("seq 1 600", "seq 1 10").replace("flock -w 150 9", "flock -w 2 9");
88120 writeFileSync(path.join(bin, "docker"), script);
89121 const env = { ...process.env, PATH: `${bin}:${process.env.PATH}` };
90122 const r = spawnSync("bash", ["-c", "docker version & sleep 0.2; docker info; echo \"second exited $?\"; wait"], { env, encoding: "utf8", timeout: 20_000 });
@@ -93,4 +125,37 @@
93125 expect(r.stderr).toContain("The Docker daemon did not start");
94126 expect(r.stderr).toContain("failed to start daemon: no cgroups");
95127 }, 30_000);
128+
129+ it("starts the registry relay and the daemon together", () => {
130+ // Two seconds each: one after the other they would take four.
131+ const bin = install({ daemonMs: 2000, relayMs: 2000 });
132+ const env = { ...process.env, PATH: `${bin}:${process.env.PATH}`, DIFFINERY_API_URL: "https://diffinery.test/api" };
133+ const started = Date.now();
134+ const r = spawnSync("bash", ["-c", "docker version"], { env, encoding: "utf8", timeout: 20_000 });
135+ const took = Date.now() - started;
136+ expect(r.status, r.stderr).toBe(0);
137+ expect(took).toBeLessThan(3500);
138+ expect(r.stderr).not.toContain("relay did not start");
139+ expect(readFileSync(path.join(dir, "dockerd.proxy"), "utf8")).toBe(`http://127.0.0.1:${relayPort}\n`);
140+ expect(readFileSync(path.join(dir, "state", "relay.host"), "utf8")).toBe("diffinery.test\n");
141+ expect(existsSync(path.join(dir, "certs.d", "diffinery.test", "ca.crt"))).toBe(true);
142+ }, 30_000);
143+
144+ it("starts the daemon again without the relay when the relay never comes up", () => {
145+ const bin = install({ relayMs: null });
146+ // Shorten the relay wait for the test.
147+ const script = readFileSync(path.join(bin, "docker"), "utf8").replace("seq 1 400", "seq 1 10");
148+ writeFileSync(path.join(bin, "docker"), script);
149+ const env = { ...process.env, PATH: `${bin}:${process.env.PATH}`, DIFFINERY_API_URL: "https://diffinery.test/api" };
150+ const r = spawnSync("bash", ["-c", "docker version"], { env, encoding: "utf8", timeout: 20_000 });
151+ expect(r.status, r.stderr).toBe(0);
152+ expect(r.stdout).toContain("docker version");
153+ expect(r.stderr).toContain("The registry relay did not start");
154+ expect(readFileSync(path.join(dir, "dockerd.proxy"), "utf8")).toBe(`http://127.0.0.1:${relayPort}\nnone\n`);
155+ const [first, second] = readFileSync(path.join(dir, "dockerd.pids"), "utf8").split("\n").filter(Boolean);
156+ // Gone (or a zombie: nothing may reap it in a container), while the new one runs.
157+ expect(running(first!)).toBe(false);
158+ expect(running(second!)).toBe(true);
159+ expect(existsSync(path.join(dir, "state", "relay.host"))).toBe(false);
160+ }, 30_000);
96161 });