Diffinery

codician/diffinerypublic

Open-source, agent-first GitHub and GitHub Actions alternative that runs on your own Cloudflare account: git hosting, pull requests, code review and drop-in GitHub Actions YAML on Workers, Durable Objects, R2 and Containers.

IndexNow: try Bing's and Yandex's endpoints when api.indexnow.org rate-limits the Worker (#259)

* IndexNow: try Bing's and Yandex's endpoints when api.indexnow.org rate-limits the Worker
test-bot authored 1 parent f454e33commit 1d8659c3cf5a75b190fe50f63d541d2efb0276f9Browse files
6 files changed+82−33
Files (6)
modifiedapps/diffinery/docs/admin/site-administration.md+2−2View file
@@ -419,11 +419,11 @@
419419 - **Every hour**, public repositories pushed since they were last submitted (their page, `/<owner>/<repo>`, as the sitemap lists it).
420420 - **Never** a private repository, a repository whose owner is suspended, or any page behind sign-in (settings, notifications, the API, sign-in pages). A repository is only submitted in the same check that found it public, so one made private in the meantime isn't sent, and it is forgotten. Removed site and docs pages are submitted once more, so search engines learn they are gone.
421421
422-How it works: the Worker's five-minute cron notices a new deploy, hashes each page's content and compares the hashes with the ones it last submitted, then sends only the differences in one request to `api.indexnow.org` (up to 10,000 URLs; more wait for the next run). A run on an older version during a deploy's rollout is ignored. Submitting happens after the deploy, in the background, so it can't fail or slow one. If IndexNow answers `429 Too Many Requests`, the pages stay queued for an hour; after another refusal (`400`, `403`, `422`) for 6 hours; if it can't be reached or fails (`5xx`, a timeout), for 15 minutes. IndexNow's `200 OK` and `202 Accepted` (key validation pending) count as submitted.
422+How it works: the Worker's five-minute cron notices a new deploy, hashes each page's content and compares the hashes with the ones it last submitted, then sends only the differences in one request (up to 10,000 URLs; more wait for the next run) to `api.indexnow.org`. Engines share what any of them receives, so when `api.indexnow.org` answers `429` or `5xx` or can't be reached, the same request goes to Bing's own endpoint (`www.bing.com/indexnow`), then Yandex's (`yandex.com/indexnow`): IndexNow limits requests per IP address, and a Worker sends from addresses it shares with other Workers. A run on an older version during a deploy's rollout is ignored. Submitting happens after the deploy, in the background, so it can't fail or slow one. If every endpoint answers `429 Too Many Requests`, the pages stay queued for an hour; after another refusal (`400`, `403`, `422`) for 6 hours; if it can't be reached or fails (`5xx`, a timeout), for 15 minutes. IndexNow's `200 OK` and `202 Accepted` (key validation pending) count as submitted.
423423
424424 The key that proves submissions come from this site is served at the site root, `/<key>.txt`. It is 32 hex characters derived from the instance's `MASTER_KEY` and `PUBLIC_URL`, so each instance has its own and nothing needs configuring. [Rotating MASTER_KEY](/docs/self-hosting/configuration#rotating-master_key) gives the instance a new key, which the next submission uses.
425425
426-**Settings → Search engines** (`/settings/admin/search-engines`) shows whether IndexNow is on and its key file, how many pages and public repositories it tracks and how many are waiting, and the newest submissions: when, how many URLs, IndexNow's answer and how long it took, any error, and the pages sent. The newest 200 submissions are kept. The same is available over the API:
426+**Settings → Search engines** (`/settings/admin/search-engines`) shows whether IndexNow is on and its key file, how many pages and public repositories it tracks and how many are waiting, and the newest submissions: when, how many URLs, which endpoint answered, its answer and how long it took, any error, and the pages sent. The newest 200 submissions are kept. The same is available over the API:
427427
428428 ```bash
429429 diffinery api GET "/diffinery/admin/indexnow?per_page=10"
addedapps/diffinery/migrations/0040_indexnow_endpoint.sql+3−0View file
@@ -1,0 +1,3 @@
1+-- The IndexNow endpoint that answered a submission (web/indexnow.ts tries the next one when an endpoint
2+-- rate-limits the Worker's shared outbound addresses or can't be reached).
3+ALTER TABLE indexnow_submissions ADD COLUMN endpoint TEXT;
modifiedapps/diffinery/src/docs/api-meta.ts+2−2View file
@@ -159,7 +159,7 @@
159159 export const API_DOCS: Record<string, OpDoc> = {
160160 // ------------------------------------------------------------ meta, users and organizations
161161 "GET /": { summary: "API root", description: "Names the API and links to these docs and the MCP endpoint.", access: "public" },
162- "GET /meta": { summary: "Instance metadata", description: "Whether this instance issues OIDC tokens to workflows, and the runner image its last deploy set (`runner_image`; each job reports the image it ran on as `runner.image`), and whether it tells search engines about changed public pages through IndexNow (`indexnow`: `enabled`, `key_location` and the time, URL count and IndexNow status of the `last_submission`; see [Site administration](/docs/admin/site-administration#search-engines)).", access: "public" },
162+ "GET /meta": { summary: "Instance metadata", description: "Whether this instance issues OIDC tokens to workflows, and the runner image its last deploy set (`runner_image`; each job reports the image it ran on as `runner.image`), and whether it tells search engines about changed public pages through IndexNow (`indexnow`: `enabled`, `key_location` and the time, URL count, IndexNow status and `endpoint` of the `last_submission`; see [Site administration](/docs/admin/site-administration#search-engines)).", access: "public" },
163163 "GET /rate_limit": {
164164 summary: "Get rate limit status",
165165 description: "For GitHub clients that check their budget before calling. It always reports a full, constant budget (15000 core requests an hour). Diffinery's real limits are per minute (2,000 requests per account, 60 new repositories, issues, pull requests, comments and the like, 300 requests per IP address without signing in) and answer 429 with `retry-after: 60`; see /docs/reference/rest-api#rate-limits.",
@@ -1653,7 +1653,7 @@
16531653 },
16541654 "GET /diffinery/admin/indexnow": {
16551655 summary: "IndexNow submissions",
1656- description: "What this instance has told search engines through IndexNow: `enabled` (and `off_reason` when it isn't), `key_location`, how many `site_pages` and public `repositories` it tracks, how many are `queued`, `retry_at` (set after IndexNow refused or couldn't be reached), `repositories_checked_at`, and the newest `submissions` (`at`, `urls`, `status` from IndexNow or null when unreachable, `ms`, `error`, and the first 50 `paths`). See [Search engines](/docs/admin/site-administration#search-engines).",
1656+ description: "What this instance has told search engines through IndexNow: `enabled` (and `off_reason` when it isn't), `key_location`, how many `site_pages` and public `repositories` it tracks, how many are `queued`, `retry_at` (set after IndexNow refused or couldn't be reached), `repositories_checked_at`, and the newest `submissions` (`at`, `urls`, `status` from IndexNow or null when unreachable, `endpoint` (the host that answered), `ms`, `error`, and the first 50 `paths`). See [Search engines](/docs/admin/site-administration#search-engines).",
16571657 access: "site-admin",
16581658 query: { per_page: ["integer", "Submissions to return (default 50, up to 200)."] },
16591659 },
modifiedapps/diffinery/src/web/account.tsx+1−1View file
@@ -1991,7 +1991,7 @@
19911991 </td>
19921992 <td>{r.urls}</td>
19931993 <td class="small">
1994- <span class={`badge${r.status === 200 || r.status === 202 ? " green" : " red"}`}>{r.status ?? "unreachable"}</span> <span class="muted">{r.ms} ms</span>
1994+ <span class={`badge${r.status === 200 || r.status === 202 ? " green" : " red"}`}>{r.status ?? "unreachable"}</span> <span class="muted">{r.endpoint ? `${r.endpoint}, ` : ""}{r.ms} ms</span>
19951995 {r.error ? <div class="mono muted">{r.error}</div> : null}
19961996 </td>
19971997 <td class="small mono" style="max-width:520px">
modifiedapps/diffinery/src/web/indexnow.ts+40−19View file
@@ -13,8 +13,9 @@
1313 * queued. Only repositories that are public at that moment, with an owner who isn't suspended, are ever
1414 * submitted (the same ones the sitemap lists), and only in the same run that saw them public: a
1515 * private repository's page, or anything behind sign-in, is never sent.
16- * - One POST to api.indexnow.org per run, at most MAX_URLS URLs (IndexNow takes 10,000). A refusal or a
17- * failure to reach IndexNow keeps the pages queued and waits before trying again; it never touches a
16+ * - One POST per run, at most MAX_URLS URLs (IndexNow takes 10,000), to api.indexnow.org, or to Bing's or
17+ * Yandex's own endpoint when it is rate-limited or down (INDEXNOW_ENDPOINTS). A refusal or a failure to
18+ * reach IndexNow keeps the pages queued and waits before trying again; it never touches a
1819 * deploy, which finished before the cron noticed it. Each submission is logged (console and
1920 * indexnow_submissions, shown on Settings → Search engines).
2021 *
@@ -30,7 +31,13 @@
3031 import { sha256 } from "../lib/util.js";
3132 import { homeContent, homeMarkdown, sitePageMarkdown, sitePages } from "./site.js";
3233
33-export const INDEXNOW_ENDPOINT = "https://api.indexnow.org/indexnow";
34+/**
35+ * Where submissions go, in order. IndexNow engines share what each one receives, so one accepted
36+ * submission reaches them all. The next endpoint is tried only when one answers 429 or 5xx or can't be
37+ * reached: rate limits are per IP address, and a Worker's outbound addresses are shared with every
38+ * other Worker, so api.indexnow.org may turn it away while an engine's own endpoint takes it.
39+ */
40+export const INDEXNOW_ENDPOINTS = ["https://api.indexnow.org/indexnow", "https://www.bing.com/indexnow", "https://yandex.com/indexnow"];
3441 /** URLs per submission: IndexNow's limit. */
3542 export const MAX_URLS = 10_000;
3643 /** Public repositories checked (and listed in the sitemap). */
@@ -198,9 +205,12 @@
198205
199206 // ---------------------------------------------------------------- submitting
200207
201-export type Submission = { at: string; urls: number; status: number | null; ms: number; error: string | null; paths: string[] };
208+export type Submission = { at: string; urls: number; status: number | null; ms: number; error: string | null; paths: string[]; endpoint: string | null };
202209
203-/** POSTs `paths` to IndexNow. Never throws. */
210+/** The host of an endpoint, as logged. */
211+const hostOf = (url: string) => new URL(url).host;
212+
213+/** POSTs `paths` to IndexNow, trying the next endpoint after a 429, a 5xx or no answer. Never throws. */
204214 export async function submit(env: Pick<Env, "PUBLIC_URL">, key: string, paths: string[], fetcher: typeof fetch = fetch): Promise<Submission> {
205215 const o = origin(env);
206216 const at = new Date().toISOString();
@@ -208,15 +218,26 @@
208218 const body = JSON.stringify({ host: new URL(o).host, key, keyLocation: keyLocation(env, key), urlList: paths.map((p) => `${o}${p}`) });
209219 let status: number | null = null;
210220 let error: string | null = null;
211- try {
212- const res = await fetcher(INDEXNOW_ENDPOINT, { method: "POST", headers: { "content-type": "application/json; charset=utf-8" }, body, signal: AbortSignal.timeout(TIMEOUT_MS) });
213- status = res.status;
214- const text = (await res.text().catch(() => "")).replace(/\s+/g, " ").trim();
215- if (status !== 200 && status !== 202) error = text.slice(0, 500) || res.statusText || `HTTP ${status}`;
216- } catch (e) {
217- error = (e instanceof Error ? `${e.name}: ${e.message}` : String(e)).slice(0, 500);
221+ let endpoint: string | null = null;
222+ const passed: string[] = [];
223+ for (const url of INDEXNOW_ENDPOINTS) {
224+ if (endpoint) passed.push(`${endpoint} ${status ?? "unreachable"}${error ? ` (${error.slice(0, 120)})` : ""}`);
225+ endpoint = hostOf(url);
226+ status = null;
227+ error = null;
228+ try {
229+ const res = await fetcher(url, { method: "POST", headers: { "content-type": "application/json; charset=utf-8" }, body, signal: AbortSignal.timeout(TIMEOUT_MS) });
230+ status = res.status;
231+ const text = (await res.text().catch(() => "")).replace(/\s+/g, " ").trim();
232+ if (status !== 200 && status !== 202) error = text.slice(0, 500) || res.statusText || `HTTP ${status}`;
233+ } catch (e) {
234+ error = (e instanceof Error ? `${e.name}: ${e.message}` : String(e)).slice(0, 500);
235+ }
236+ if (status !== null && status !== 429 && status < 500) break;
218237 }
219- return { at, urls: paths.length, status, ms: Date.now() - started, error, paths };
238+ // What the endpoints passed over said, then the last one's own error.
239+ if (passed.length) error = [`after ${passed.join("; ")}`, error].filter(Boolean).join(": ");
240+ return { at, urls: paths.length, status, ms: Date.now() - started, error: error?.slice(0, 1000) ?? null, paths, endpoint };
220241 }
221242
222243 const accepted = (s: Submission) => s.status === 200 || s.status === 202;
@@ -249,9 +270,9 @@
249270 }
250271 const batch = all.slice(0, MAX_URLS);
251272 const s = await submit(env, await key, batch.map((r) => r.path), fetcher);
252- console.log(`indexnow: ${s.urls} URLs, ${s.status ?? "unreachable"} in ${s.ms} ms${s.error ? `: ${s.error}` : ""}`);
273+ console.log(`indexnow: ${s.urls} URLs, ${s.endpoint} ${s.status ?? "unreachable"} in ${s.ms} ms${s.error ? `: ${s.error}` : ""}`);
253274 const stmts: D1PreparedStatement[] = [
254- db.prepare("INSERT INTO indexnow_submissions (at, urls, status, ms, error, paths) VALUES (?, ?, ?, ?, ?, ?)").bind(s.at, s.urls, s.status, s.ms, s.error, JSON.stringify(s.paths.slice(0, 50))),
275+ db.prepare("INSERT INTO indexnow_submissions (at, urls, status, ms, error, paths, endpoint) VALUES (?, ?, ?, ?, ?, ?, ?)").bind(s.at, s.urls, s.status, s.ms, s.error, JSON.stringify(s.paths.slice(0, 50)), s.endpoint),
255276 db.prepare("DELETE FROM indexnow_submissions WHERE id <= (SELECT id FROM indexnow_submissions ORDER BY id DESC LIMIT 1 OFFSET ?)").bind(KEEP_SUBMISSIONS),
256277 ];
257278 if (accepted(s)) {
@@ -308,16 +329,16 @@
308329 }
309330
310331 export async function lastSubmissions(db: D1Database, limit: number): Promise<Submission[]> {
311- const rows = (await db.prepare("SELECT at, urls, status, ms, error, paths FROM indexnow_submissions ORDER BY id DESC LIMIT ?").bind(limit).all<Omit<Submission, "paths"> & { paths: string }>()).results;
332+ const rows = (await db.prepare("SELECT at, urls, status, ms, error, paths, endpoint FROM indexnow_submissions ORDER BY id DESC LIMIT ?").bind(limit).all<Omit<Submission, "paths"> & { paths: string }>()).results;
312333 return rows.map((r) => ({ ...r, paths: JSON.parse(r.paths) as string[] }));
313334 }
314335
315336 /** For GET /api/v3/meta: whether IndexNow is on, its key file, and how the last submission went (no URLs). */
316-export async function indexNowMeta(env: IndexNowEnv): Promise<{ enabled: boolean; key_location: string | null; last_submission: { at: string; urls: number; status: number | null } | null }> {
337+export async function indexNowMeta(env: IndexNowEnv): Promise<{ enabled: boolean; key_location: string | null; last_submission: { at: string; urls: number; status: number | null; endpoint: string | null } | null }> {
317338 const key = indexNowKey(env);
318339 if (!key) return { enabled: false, key_location: null, last_submission: null };
319- const last = await env.DB.prepare("SELECT at, urls, status FROM indexnow_submissions ORDER BY id DESC LIMIT 1")
320- .first<{ at: string; urls: number; status: number | null }>()
340+ const last = await env.DB.prepare("SELECT at, urls, status, endpoint FROM indexnow_submissions ORDER BY id DESC LIMIT 1")
341+ .first<{ at: string; urls: number; status: number | null; endpoint: string | null }>()
321342 .catch(() => null);
322343 return { enabled: true, key_location: keyLocation(env, await key), last_submission: last ?? null };
323344 }
modifiedapps/diffinery/test/unit/indexnow.test.ts+34−9View file
@@ -10,7 +10,7 @@
1010 import type { Repo } from "../../src/lib/directory.js";
1111 import { docsSitemapPaths } from "../../src/docs/web.js";
1212 import { siteSitemapEntries } from "../../src/web/site.js";
13-import { indexNowCron, indexNowKey, indexNowOff, indexNowStatus, MAX_URLS, RETRY_MS, sitePageContents } from "../../src/web/indexnow.js";
13+import { INDEXNOW_ENDPOINTS, indexNowCron, indexNowKey, indexNowOff, indexNowStatus, MAX_URLS, RETRY_MS, sitePageContents } from "../../src/web/indexnow.js";
1414 import { testD1 } from "./stubs/d1.js";
1515
1616 app.route("/", web);
@@ -21,16 +21,18 @@
2121
2222 type Sent = { host: string; key: string; keyLocation: string; urlList: string[] };
2323
24-function setup(o: { status?: number | (() => number); version?: string } = {}) {
24+function setup(o: { status?: number | ((url: string) => number); version?: string } = {}) {
2525 const db = testD1();
2626 const env = { PUBLIC_URL: ORIGIN, MASTER_KEY, DB: db, CF_VERSION_METADATA: o.version ? { id: "v", tag: "", timestamp: o.version } : undefined } as unknown as Env;
2727 const sent: Sent[] = [];
28+ const urls: string[] = [];
2829 let status = o.status ?? 200;
2930 const fetcher = (async (url: string, init: RequestInit) => {
30- expect(url).toBe("https://api.indexnow.org/indexnow");
31+ expect(INDEXNOW_ENDPOINTS).toContain(url);
3132 expect(new Headers(init.headers).get("content-type")).toBe("application/json; charset=utf-8");
33+ urls.push(url);
3234 sent.push(JSON.parse(init.body as string) as Sent);
33- const s = typeof status === "function" ? status() : status;
35+ const s = typeof status === "function" ? status(url) : status;
3436 return new Response(s === 200 ? "" : "nope", { status: s });
3537 }) as unknown as typeof fetch;
3638 let repos: Array<Partial<Repo>> = [];
@@ -39,11 +41,12 @@
3941 env,
4042 db,
4143 sent,
44+ urls,
4245 run: () => indexNowCron(env as never, dir, fetcher),
4346 setRepos: (r: Array<Partial<Repo>>) => (repos = r),
4447 /** A new version, as a deploy makes (its content is the same build, so tests change what was last submitted instead). */
4548 deploy: (timestamp: string) => ((env as { CF_VERSION_METADATA?: WorkerVersionMetadata }).CF_VERSION_METADATA = { id: `v-${timestamp}`, tag: "", timestamp }),
46- setStatus: (s: number) => (status = s),
49+ setStatus: (s: number | ((url: string) => number)) => (status = s),
4750 state: (k: string) => db.sqlite.prepare("SELECT v FROM indexnow_state WHERE k = ?").get(k) as { v: string } | undefined,
4851 /** Moves time forward for the cron: every stored time goes back by `ms`. */
4952 age: (ms: number) => {
@@ -184,12 +187,14 @@
184187 it("keeps pages queued after a refusal and waits before trying again", async () => {
185188 t.setStatus(429);
186189 const s = await t.run();
187- expect(s).toMatchObject({ status: 429, error: "nope" });
190+ // Every endpoint was tried once, in order.
191+ expect(t.urls).toEqual(INDEXNOW_ENDPOINTS);
192+ expect(s).toMatchObject({ status: 429, endpoint: "yandex.com", error: "after api.indexnow.org 429 (nope); www.bing.com 429 (nope): nope" });
188193 const retry = Date.parse(t.state("retry_at")!.v);
189194 expect(retry - Date.now()).toBeGreaterThan(RETRY_MS.rateLimited - 60_000);
190195 // Waiting: nothing sent.
191196 expect(await t.run()).toBeNull();
192- expect(t.sent).toHaveLength(1);
197+ expect(t.sent).toHaveLength(3);
193198 // Once the wait is over, the same pages go again.
194199 t.setStatus(202);
195200 t.age(RETRY_MS.rateLimited);
@@ -198,15 +203,35 @@
198203 expect(await t.run()).toBeNull();
199204 const st = await indexNowStatus(t.env as never);
200205 expect(st).toMatchObject({ off: null, queued: 0, retryAt: null });
201- expect(st.submissions.map((x) => x.status)).toEqual([202, 429]);
206+ expect(st.submissions.map((x) => [x.status, x.endpoint])).toEqual([
207+ [202, "api.indexnow.org"],
208+ [429, "yandex.com"],
209+ ]);
202210 });
203211
212+ it("tries the next endpoint when one rate-limits or fails, but not after a refusal", async () => {
213+ t.setStatus((url) => (url.includes("api.indexnow.org") ? 429 : 200));
214+ const s = await t.run();
215+ expect(s).toMatchObject({ status: 200, endpoint: "www.bing.com", error: "after api.indexnow.org 429 (nope)" });
216+ expect(t.urls).toEqual(INDEXNOW_ENDPOINTS.slice(0, 2));
217+ // The pages count as submitted.
218+ expect(await t.run()).toBeNull();
219+ // A 403 (key not valid) or 422 is the same everywhere: no other endpoint is tried.
220+ t.db.sqlite.prepare("UPDATE indexnow_urls SET sent_hash = 'old' WHERE path = '/features'").run();
221+ t.db.sqlite.prepare("INSERT INTO indexnow_state (k, v) VALUES ('retry_at', '2000-01-01T00:00:00Z')").run();
222+ t.setStatus(403);
223+ expect(await t.run()).toMatchObject({ status: 403, endpoint: "api.indexnow.org", error: "nope" });
224+ expect(t.urls).toHaveLength(3);
225+ expect(Date.parse(t.state("retry_at")!.v) - Date.now()).toBeGreaterThan(RETRY_MS.refused - 60_000);
226+ });
227+
204228 it("never throws when IndexNow can't be reached, and tries again later", async () => {
205229 const fetcher = (async () => {
206230 throw new TypeError("network down");
207231 }) as unknown as typeof fetch;
208232 const s = await indexNowCron(t.env as never, { publicRepos: async () => [] }, fetcher);
209- expect(s).toMatchObject({ status: null, error: "TypeError: network down" });
233+ expect(s).toMatchObject({ status: null, endpoint: "yandex.com" });
234+ expect(s!.error).toMatch(/^after api.indexnow.org unreachable \(TypeError: network down\); www.bing.com unreachable .*: TypeError: network down$/);
210235 expect(Date.parse(t.state("retry_at")!.v)).toBeGreaterThan(Date.now());
211236 // A broken database is logged, not thrown.
212237 const broken = { ...t.env, DB: { prepare: () => { throw new Error("D1 down"); } } };