Diffinery

codician/diffinerypublic

Open-source, agent-first GitHub and GitHub Actions alternative that runs on your own Cloudflare account: git hosting, pull requests, code review and drop-in GitHub Actions YAML on Workers, Durable Objects, R2 and Containers.

IndexNow: tell search engines about new, changed and removed public pages after every deploy (#258)

* IndexNow: tell search engines about new, changed and removed public pages after every deploy
test-bot authored 1 parent 4d9ceaacommit 93b89d65cca1343a4233918fbaed2fc70aa7c709Browse files
16 files changed+762−6
Files (16)
modifiedapps/diffinery/docs/admin/site-administration.md+25−0View file
@@ -18,6 +18,7 @@
1818 | **Kill switches** | `/settings/admin/kill-switches` | Stop Actions and agent sandboxes for a repository, an account or the whole instance |
1919 | **Security reports** | `/settings/admin/security-reports` | Private vulnerability reports from the report form, to read, triage and resolve |
2020 | **Status** | `/settings/admin/status` | Incidents on the public status page, to open, update and resolve |
21+| **Search engines** | `/settings/admin/search-engines` | What Diffinery told search engines through IndexNow about new, changed and removed public pages |
2122 | **Audit log** | `/settings/audit` | The most recent security-relevant events |
2223
2324 Their **Settings → Profile** page shows **Site admin** as yes, and they have an **admin** badge in the users list.
@@ -408,6 +409,30 @@
408409
409410 To anyone else these endpoints answer `404`. The status page runs in the instance's own Worker, so it is down whenever the whole Worker is: watch `/healthz` with an external uptime monitor too (see [Monitoring from outside](/docs/self-hosting/status-page#monitoring-from-outside)).
410411
412+## Search engines
413+
414+Diffinery tells search engines about public pages that are added, change or go away through [IndexNow](https://www.indexnow.org), so Bing, Yandex, Naver, Seznam, Yep and the other engines that share IndexNow submissions recrawl them within minutes instead of whenever they next visit. Nobody needs to do anything: it runs by itself after every deploy.
415+
416+What it covers:
417+
418+- **After each deploy**, the pages the deploy added, changed or removed: the home page, the product, comparison and company pages (`/features`, `/compare/github`, ...), every docs page, each of those pages' Markdown version (`/index.md`, `/compare/github.md`, `/docs/quickstart.md`, ...), `/llms.txt`, `/llms-full.txt`, and `/early-access` while the instance is invite-only. These are the pages in `/sitemap.xml` plus their Markdown versions. A deploy that changes no page's content submits nothing.
419+- **Every hour**, public repositories pushed since they were last submitted (their page, `/<owner>/<repo>`, as the sitemap lists it).
420+- **Never** a private repository, a repository whose owner is suspended, or any page behind sign-in (settings, notifications, the API, sign-in pages). A repository is only submitted in the same check that found it public, so one made private in the meantime isn't sent, and it is forgotten. Removed site and docs pages are submitted once more, so search engines learn they are gone.
421+
422+How it works: the Worker's five-minute cron notices a new deploy, hashes each page's content and compares the hashes with the ones it last submitted, then sends only the differences in one request to `api.indexnow.org` (up to 10,000 URLs; more wait for the next run). A run on an older version during a deploy's rollout is ignored. Submitting happens after the deploy, in the background, so it can't fail or slow one. If IndexNow answers `429 Too Many Requests`, the pages stay queued for an hour; after another refusal (`400`, `403`, `422`) for 6 hours; if it can't be reached or fails (`5xx`, a timeout), for 15 minutes. IndexNow's `200 OK` and `202 Accepted` (key validation pending) count as submitted.
423+
424+The key that proves submissions come from this site is served at the site root, `/<key>.txt`. It is 32 hex characters derived from the instance's `MASTER_KEY` and `PUBLIC_URL`, so each instance has its own and nothing needs configuring. [Rotating MASTER_KEY](/docs/self-hosting/configuration#rotating-master_key) gives the instance a new key, which the next submission uses.
425+
426+**Settings → Search engines** (`/settings/admin/search-engines`) shows whether IndexNow is on and its key file, how many pages and public repositories it tracks and how many are waiting, and the newest submissions: when, how many URLs, IndexNow's answer and how long it took, any error, and the pages sent. The newest 200 submissions are kept. The same is available over the API:
427+
428+```bash
429+diffinery api GET "/diffinery/admin/indexnow?per_page=10"
430+```
431+
432+Anyone can see whether it is on and how the last submission went, without the pages, in `GET /api/v3/meta` (`indexnow`). Each submission is also logged in the Worker's logs as `indexnow: <n> URLs, <status> in <ms> ms`.
433+
434+IndexNow is off when `INDEXNOW` is `off`, when `PUBLIC_URL` isn't a public `https` origin (`localhost`, an IP address or `http`, as in local development), or when `MASTER_KEY` isn't set. See [Configuration](/docs/self-hosting/configuration#worker-variables).
435+
411436 ## Workflow scheduler
412437
413438 Scheduled workflows are started by Diffinery's own timer, every 5 minutes, which also catches up checks that came late (see [Scheduled workflows](/docs/actions/events#schedule)). When a scheduled run doesn't start, read the scheduler's state to tell a timer that stopped from a run that failed to start:
modifiedapps/diffinery/docs/repositories/overview.md+1−1View file
@@ -114,7 +114,7 @@
114114 | **Private** (default) | The owner, the organization's members (for organization repositories, unless the organization's base permission is **none**), collaborators, and site administrators. |
115115 | **Public** | Anyone, including people who are not signed in. Signed-out visitors can browse the code, and `git clone` works without credentials. Every signed-in user gets **read** access. |
116116
117-Writing to a public repository still needs a signed-in account (or a token) with a role that allows it. People who aren't signed in can read a public repository but can't change anything in it: opening an issue, commenting, reacting, watching and pushing all need an account, and the API answers such requests with `401 authentication required`. An anonymous `git push` gets a `401` with a `WWW-Authenticate` challenge, so Git asks for your credentials (or uses its credential helper) and tries again with them. A `read`-scoped token can read too, but every write with it gets `403 this token needs the write scope`. Only public repositories are indexed by search engines; private pages carry a `noindex` tag.
117+Writing to a public repository still needs a signed-in account (or a token) with a role that allows it. People who aren't signed in can read a public repository but can't change anything in it: opening an issue, commenting, reacting, watching and pushing all need an account, and the API answers such requests with `401 authentication required`. An anonymous `git push` gets a `401` with a `WWW-Authenticate` challenge, so Git asks for your credentials (or uses its credential helper) and tries again with them. A `read`-scoped token can read too, but every write with it gets `403 this token needs the write scope`. Only public repositories are indexed by search engines; private pages carry a `noindex` tag. A public repository's page is listed in the sitemap, and search engines are told about it through IndexNow within the hour after a push (see [Search engines](/docs/admin/site-administration#search-engines)); a private one never is.
118118
119119 For people without access, a private repository looks like it does not exist: the web UI, Git and the API answer `404` (`repository not found`) rather than `403`, so the name is not revealed. Signed-out requests can't tell a private repository from a missing one either: Git, Git LFS and the web UI answer both with `401` (`authentication required`), so Git asks for credentials and the web UI offers sign-in, and the REST API answers both with `404`, as GitHub does. A signed-out write gets `401` whether or not the repository exists.
120120
modifiedapps/diffinery/docs/self-hosting/architecture.md+2−1View file
@@ -24,7 +24,7 @@
2424 | The Git transfer of a GitHub import | Containers (`ImportContainer`, one container per transfer) | `IMPORTER` |
2525 | Scheduled workflows | Durable Object alarm, every 5 minutes (`SchedulerDO`, one instance) | `SCHEDULER` |
2626 | Joining large artifact and cache uploads | Durable Objects, one per step (`ComposeDO`) | `COMPOSE` |
27-| Token expiry reminders, email log pruning, registry cleanup, expiry of draft imports and of account exports, a check that the scheduler's alarm is set | Cron Triggers, every 5 minutes | `scheduled` handler |
27+| Token expiry reminders, email log pruning, registry cleanup, expiry of draft imports and of account exports, a check that the scheduler's alarm is set, IndexNow submissions | Cron Triggers, every 5 minutes | `scheduled` handler |
2828 | Live run logs and pull request updates in the browser | Durable Object WebSockets with hibernation | Through `REPO` |
2929 | Outgoing email | Email Sending | `EMAIL` |
3030 | Email-based account recovery and first admin sign-in | Cloudflare Access, on `/auth/access` only | `ACCESS_TEAM_DOMAIN`, `ACCESS_AUD` |
@@ -200,6 +200,7 @@
200200 2. Sets the workflow scheduler's alarm if it isn't set.
201201 3. Sends token expiry reminders and, once a day, prunes the email log.
202202 4. Cancels draft organization imports (started in the web form, repositories never chosen) that are more than 7 days old, with the error `draft expired`, and erases their GitHub token.
203+5. Tells search engines through IndexNow about public pages a deploy added, changed or removed, and, hourly, about public repositories pushed since (see [Search engines](/docs/admin/site-administration#search-engines)). It keeps its state in D1 (`indexnow_urls`, `indexnow_submissions`, `indexnow_state`).
203204
204205 ## Live updates
205206
modifiedapps/diffinery/docs/self-hosting/configuration.md+2−0View file
@@ -43,6 +43,7 @@
4343 | `EGRESS` | Not set | No | `web` limits hosted Actions jobs and agent sandboxes (including tasks, reviews and terminals) to HTTP and HTTPS (ports 80 and 443) and DNS. Any other value, or none, leaves them open to the whole internet. Accounts run by a site administrator stay open. See [EGRESS](#egress). |
4444 | `RUNNER_IMAGE` | Set by the deploy | No | The runner image's name and tag (for example `diffinery-runner:96dbbe31790f`), as jobs report it in `runner.image`. `scripts/render-config.mjs` sets it when the deploy uses a prebuilt runner image, and `GET /api/v3/meta` returns it as `runner_image` (`null` when not set). Don't set it by hand. |
4545 | `SECURITY_CONTACT` | Not set | No | More places people can report a vulnerability in your instance, listed in `/.well-known/security.txt` after the instance's own report form (`/security/report`): an email address or an `https://` URL, several separated by commas. See [Security contact](#security-contact). |
46+| `INDEXNOW` | Not set | No | `off` stops telling search engines about new, changed and removed public pages through IndexNow. Not set, or anything else, it is on whenever `PUBLIC_URL` is a public `https` origin and `MASTER_KEY` is set. Turn it off for an instance search engines can't reach. See [Search engines](/docs/admin/site-administration#search-engines). |
4647 | `TERMINALS` | Not set | No | `off` switches [terminals](/docs/agents/terminal) off: nobody can open one, and open ones end at their next check, within a minute. Anything else, or not set, leaves them on. |
4748
4849 When either `ACCESS_TEAM_DOMAIN` or `ACCESS_AUD` is empty, email sign-in at `/auth/access` is off: the page answers "Account recovery by email is not set up on this instance. Ask an administrator for a sign-in link.", and the sign-in page doesn't offer **Recover your account by email**. Passkeys, invitations and admin sign-in links keep working, but the first administrator can't sign in, because email sign-in is how an account in `BOOTSTRAP_ADMINS` is created.
@@ -245,6 +246,7 @@
245246 | `workers_dev` | `false` | The instance is only served on your route. |
246247 | `routes` | `[{ "pattern": "diffinery.com/*", "zone_name": "diffinery.com" }]` | A zone route, not a Custom Domain. The hostname's DNS record must be proxied. On deploy, `scripts/render-config.mjs` replaces it with one route, `<DIFFINERY_DOMAIN>/*` in the zone `DIFFINERY_ZONE` (by default the hostname's last two labels), so you don't edit it by hand. |
247248 | `observability` | Enabled, `head_sampling_rate: 1` | Every request is logged to Workers Logs. |
249+| `version_metadata` | Binding `CF_VERSION_METADATA` | The deployed version's id and time. [IndexNow](/docs/admin/site-administration#search-engines) checks the site's pages once for each new version and ignores older versions still running during a rollout. Optional: without it, it checks whenever the pages' content differs from what it last submitted. |
248250 | `upload_source_maps` | `true` | Readable stack traces in the logs. |
249251 | `limits.cpu_ms` | `300000` | Up to 5 minutes of CPU time per request (large pushes, diffs and merges). |
250252 | `containers` | `RunnerContainer`: `standard-4`, `max_instances: 100`; `RunnerContainerMedium`, `RunnerContainerSmall`, `RunnerContainerTiny`: `standard-3`, `standard-1`, `basic`, `max_instances: 100` each (smaller hosted runners; remove them and their `RUNNER_*` bindings to run every job on `standard-4`); `SandboxContainer`: `standard-1`, `max_instances: 50`; `ImportContainer`: `standard-4`, `max_instances: 20`; `rollout_active_grace_period`: 23400 for runners and 25200 for imports (longer than a job or transfer can run, so a deploy never replaces a running one), 3600 for sandboxes; `rollout_step_percentage: 100`, so containers started after a deploy get the new image | All three build `containers/runner/Dockerfile` with `containers/runner` as context. Raise `max_instances` for more concurrent jobs, sandboxes or import transfers. |
addedapps/diffinery/migrations/0039_indexnow.sql+34−0View file
@@ -1,0 +1,34 @@
1+-- IndexNow (lib/indexnow.ts): the public pages Diffinery tells search engines about when they are added,
2+-- change or go away, and what it last told them.
3+--
4+-- One row per public URL path. `kind` is 'site' for the pages built into the Worker (the home page, the
5+-- product, comparison and company pages, the docs, their .md versions, llms.txt) or 'repo' for a public
6+-- repository's page. `hash` is the page's content hash ('site'), its last push time ('repo'), or '' once a
7+-- site page is gone. `sent_hash` is the hash last submitted: a row waits to be submitted while it differs.
8+CREATE TABLE IF NOT EXISTS indexnow_urls (
9+ path TEXT PRIMARY KEY,
10+ kind TEXT NOT NULL,
11+ hash TEXT NOT NULL,
12+ sent_hash TEXT,
13+ sent_at TEXT
14+);
15+CREATE INDEX IF NOT EXISTS indexnow_urls_kind ON indexnow_urls(kind);
16+
17+-- Each submission to IndexNow, newest kept (Settings → Search engines). `status` is IndexNow's HTTP status,
18+-- NULL when it couldn't be reached; `paths` is JSON, the first paths submitted.
19+CREATE TABLE IF NOT EXISTS indexnow_submissions (
20+ id INTEGER PRIMARY KEY AUTOINCREMENT,
21+ at TEXT NOT NULL,
22+ urls INTEGER NOT NULL,
23+ status INTEGER,
24+ ms INTEGER NOT NULL,
25+ error TEXT,
26+ paths TEXT NOT NULL
27+);
28+
29+-- Small state: the site's content fingerprint and the deploy it came from, when repositories were last
30+-- checked, and when to try again after IndexNow refused or couldn't be reached.
31+CREATE TABLE IF NOT EXISTS indexnow_state (
32+ k TEXT PRIMARY KEY,
33+ v TEXT NOT NULL
34+);
modifiedapps/diffinery/src/app.ts+3−0View file
@@ -272,6 +272,9 @@
272272 // records (lib/retention.ts): a bounded batch each hour, which keeps up with a busy instance.
273273 (at.getUTCMinutes() < 5 ? pruneAuditLog(env.DB, at) : Promise.resolve(0)).catch((e) => console.error("audit log pruning failed", e)),
274274 (at.getUTCMinutes() < 5 ? pruneExpired(env.DB, at) : Promise.resolve(null)).catch((e) => console.error("retention pruning failed", e)),
275+ // Tell search engines about public pages a deploy added, changed or removed, and public repositories pushed since (web/indexnow.ts). Never throws.
276+ // Imported here: web/indexnow.ts reaches the docs, whose API reference imports this module.
277+ import("./web/indexnow.js").then((m) => m.indexNowCron(env, new Directory(env))).catch((e) => console.error("indexnow failed", e)),
275278 // CLI sign-ins a day past their expiry (auth/cli-login.ts).
276279 new CliLogins(env).prune().catch((e) => console.error("cli sign-in pruning failed", e)),
277280 // Spent OAuth codes, expired connected-app tokens, old connections and unused registrations (auth/oauth.ts).
modifiedapps/diffinery/src/docs/api-meta.ts+7−1View file
@@ -159,7 +159,7 @@
159159 export const API_DOCS: Record<string, OpDoc> = {
160160 // ------------------------------------------------------------ meta, users and organizations
161161 "GET /": { summary: "API root", description: "Names the API and links to these docs and the MCP endpoint.", access: "public" },
162- "GET /meta": { summary: "Instance metadata", description: "Whether this instance issues OIDC tokens to workflows, and the runner image its last deploy set (`runner_image`; each job reports the image it ran on as `runner.image`).", access: "public" },
162+ "GET /meta": { summary: "Instance metadata", description: "Whether this instance issues OIDC tokens to workflows, and the runner image its last deploy set (`runner_image`; each job reports the image it ran on as `runner.image`), and whether it tells search engines about changed public pages through IndexNow (`indexnow`: `enabled`, `key_location` and the time, URL count and IndexNow status of the `last_submission`; see [Site administration](/docs/admin/site-administration#search-engines)).", access: "public" },
163163 "GET /rate_limit": {
164164 summary: "Get rate limit status",
165165 description: "For GitHub clients that check their budget before calling. It always reports a full, constant budget (15000 core requests an hour). Diffinery's real limits are per minute (2,000 requests per account, 60 new repositories, issues, pull requests, comments and the like, 300 requests per IP address without signing in) and answer 429 with `retry-after: 60`; see /docs/reference/rest-api#rate-limits.",
@@ -1651,6 +1651,12 @@
16511651 description: "`tick_minutes`, `last_tick_at` (when the last check finished), `covered_until` (the last minute whose schedules were started) and `next_tick_at` (null when the timer isn't set). Tells a scheduler that stopped ticking from a run that failed to start.",
16521652 access: "site-admin",
16531653 },
1654+ "GET /diffinery/admin/indexnow": {
1655+ summary: "IndexNow submissions",
1656+ description: "What this instance has told search engines through IndexNow: `enabled` (and `off_reason` when it isn't), `key_location`, how many `site_pages` and public `repositories` it tracks, how many are `queued`, `retry_at` (set after IndexNow refused or couldn't be reached), `repositories_checked_at`, and the newest `submissions` (`at`, `urls`, `status` from IndexNow or null when unreachable, `ms`, `error`, and the first 50 `paths`). See [Search engines](/docs/admin/site-administration#search-engines).",
1657+ access: "site-admin",
1658+ query: { per_page: ["integer", "Submissions to return (default 50, up to 200)."] },
1659+ },
16541660 "GET /diffinery/admin/errors": {
16551661 summary: "List unexpected server errors",
16561662 description: "The newest answers with a 5xx status, newest first: `id` (the `request_id` the answer carried, also in its `x-request-id` header), `at`, `method`, `route`, `path`, `status`, `message` and `stack`. The newest 500 are kept. No request bodies, query strings or headers are kept, and tokens in a message are masked.",
modifiedapps/diffinery/src/env.ts+7−0View file
@@ -101,6 +101,13 @@
101101 ARTIFACT_QUOTA_BYTES?: string;
102102 /** The runner image the deploy pointed containers at, as jobs report it (scripts/render-config.mjs). */
103103 RUNNER_IMAGE?: string;
104+ /**
105+ * "off" stops telling search engines about new and changed public pages through IndexNow (web/indexnow.ts).
106+ * Unset, it is on whenever PUBLIC_URL is a public https origin and MASTER_KEY is set.
107+ */
108+ INDEXNOW?: string;
109+ /** The deployed version (wrangler.jsonc version_metadata): IndexNow checks the site's pages once per new version. */
110+ CF_VERSION_METADATA?: WorkerVersionMetadata;
104111 /** Local development only: act as this email when Access is not configured. */
105112 DEV_AUTH?: string;
106113 /**
modifiedapps/diffinery/src/http/api.ts+10−1View file
@@ -69,7 +69,9 @@
6969
7070 export const apiRootJson = (c: AppContext) => c.json({ name: "Diffinery", api: "v3", docs: `${c.env.PUBLIC_URL}/docs/api`, mcp: `${c.env.PUBLIC_URL}/mcp`, graphql: `${c.env.PUBLIC_URL}/api/graphql` });
7171 api.get("/", apiRootJson);
72-api.get("/meta", (c) => c.json({ name: "Diffinery", oidc: !!c.env.OIDC_PRIVATE_KEY, runner_image: c.env.RUNNER_IMAGE ?? null }));
72+// Imported when used: web/indexnow.ts reaches the docs, whose API reference imports this module.
73+const indexNow = () => import("../web/indexnow.js");
74+api.get("/meta", async (c) => c.json({ name: "Diffinery", oidc: !!c.env.OIDC_PRIVATE_KEY, runner_image: c.env.RUNNER_IMAGE ?? null, indexnow: await (await indexNow()).indexNowMeta(c.env) }));
7375 /** Site admin: the newest unexpected server errors (lib/server-errors.ts), and one by the request id its 5xx answer carried. */
7476 api.get("/diffinery/admin/errors", async (c) => {
7577 if (!requireAccount(c).site_admin) throw new HttpError(403, "site administrators only");
@@ -82,6 +84,13 @@
8284 if (!e) throw new HttpError(404, "no server error with that request id (only the newest 500 are kept)");
8385 return c.json(e);
8486 });
87+/** Site admin: what IndexNow has told search engines (web/indexnow.ts). */
88+api.get("/diffinery/admin/indexnow", async (c) => {
89+ if (!requireAccount(c).site_admin) throw new HttpError(403, "site administrators only");
90+ const n = Number(c.req.query("per_page") ?? 50);
91+ const s = await (await indexNow()).indexNowStatus(c.env, Number.isFinite(n) ? Math.max(1, Math.min(200, Math.floor(n))) : 50);
92+ return c.json({ enabled: !s.off, off_reason: s.off, key_location: s.keyLocation, site_pages: s.site, repositories: s.repos, queued: s.queued, retry_at: s.retryAt, repositories_checked_at: s.reposCheckedAt, submissions: s.submissions });
93+});
8594 /** Site admin: the workflow scheduler's state, to tell a scheduler that stopped ticking from a run that failed to start. */
8695 api.get("/diffinery/admin/scheduler", async (c) => {
8796 if (!requireAccount(c).site_admin) throw new HttpError(403, "site administrators only");
modifiedapps/diffinery/src/web/account.tsx+90−2View file
@@ -26,6 +26,7 @@
2626 import { TodayView } from "../agentic/web.js";
2727 import { landingPage } from "./site.js";
2828 import { serverErrors } from "../lib/server-errors.js";
29+import { indexNowStatus } from "./indexnow.js";
2930 import { ComputePausedBanner, KillSwitchesPage } from "./kill-switches.js";
3031 import { changePause, changeSuspension, cleanReason, type Stopped } from "../lib/killswitch.js";
3132 import { blockList, cleanResolution, openReportCount, report, reports, setReportState, suspend, suspendReason, unsuspend, visibleTo } from "../lib/moderation.js";
@@ -615,9 +616,9 @@
615616
616617 // ---------------------------------------------------------------- /settings
617618
618-export type SettingsTab = "profile" | "account" | "appearance" | "security" | "tokens" | "agents" | "model-access" | "orgs" | "invitations" | "notifications" | "email-address" | "blocked" | "linked" | "admin" | "reports" | "email" | "audit" | "kill-switches" | "errors" | "security-reports" | "status" | "early-access";
619+export type SettingsTab = "profile" | "account" | "appearance" | "security" | "tokens" | "agents" | "model-access" | "orgs" | "invitations" | "notifications" | "email-address" | "blocked" | "linked" | "admin" | "reports" | "email" | "audit" | "kill-switches" | "errors" | "security-reports" | "status" | "early-access" | "search-engines";
619620 /** The site administration pages: they show when hosted compute is paused. */
620-export const ADMIN_TABS: readonly SettingsTab[] = ["admin", "early-access", "reports", "email", "audit", "kill-switches", "errors", "security-reports", "status"];
621+export const ADMIN_TABS: readonly SettingsTab[] = ["admin", "early-access", "reports", "email", "audit", "kill-switches", "errors", "security-reports", "status", "search-engines"];
621622
622623 /** The number of confirmed early access requests waiting, next to Early access in the sidebar (lib/early-access.ts). */
623624 const EarlyAccessWaiting = async ({ db }: { db?: D1Database }) => {
@@ -725,6 +726,9 @@
725726 <a href="/settings/admin/status" class={tab === "status" ? "on" : ""}>
726727 Status
727728 </a>
729+ <a href="/settings/admin/search-engines" class={tab === "search-engines" ? "on" : ""}>
730+ Search engines
731+ </a>
728732 <a href="/settings/audit" class={tab === "audit" ? "on" : ""}>
729733 Audit log
730734 </a>
@@ -1844,6 +1848,10 @@
18441848 });
18451849
18461850 // Unexpected server errors (lib/server-errors.ts); ?id= finds the one a 5xx answer's request ID names.
1851+ web.get("/settings/admin/search-engines", async (c) => {
1852+ const me = requireSiteAdmin(c);
1853+ return page(c, me, "search-engines", "Search engines", <SearchEngines s={await indexNowStatus(c.env)} />);
1854+ });
18471855 web.get("/settings/admin/errors", async (c) => {
18481856 const me = requireSiteAdmin(c);
18491857 const id = c.req.query("id")?.trim() || undefined;
@@ -1924,6 +1932,86 @@
19241932 });
19251933 }
19261934
1935+// ---------------------------------------------------------------- site administration: search engines
1936+
1937+const SearchEngines: FC<{ s: Awaited<ReturnType<typeof indexNowStatus>> }> = ({ s }) => (
1938+ <>
1939+ <h1>Search engines</h1>
1940+ <p class="muted">
1941+ Diffinery tells search engines through <a href="https://www.indexnow.org">IndexNow</a> when public pages are added, change or go away: after each deploy, the site and docs pages it changed (and their Markdown versions and llms.txt), and every hour, public repositories pushed since. Private repositories and pages behind sign-in are never sent. See{" "}
1942+ <a href="/docs/admin/site-administration#search-engines">the docs</a>.
1943+ </p>
1944+ <div class="box">
1945+ <div class="body">
1946+ {s.off ? (
1947+ <p>
1948+ <span class="badge">off</span> {s.off}.
1949+ </p>
1950+ ) : (
1951+ <p>
1952+ <span class="badge green">on</span> Key file:{" "}
1953+ <a class="mono" href={s.keyLocation!}>
1954+ {s.keyLocation}
1955+ </a>
1956+ </p>
1957+ )}
1958+ <p class="small muted" style="margin:0">
1959+ Tracking {s.site} site and docs pages and {s.repos} public repositories; {s.queued} waiting to be submitted.
1960+ {s.retryAt && Date.parse(s.retryAt) > Date.now() ? (
1961+ <>
1962+ {" "}
1963+ Trying again <Time at={s.retryAt} />.
1964+ </>
1965+ ) : null}
1966+ {s.reposCheckedAt ? (
1967+ <>
1968+ {" "}
1969+ Repositories checked <Time at={s.reposCheckedAt} />.
1970+ </>
1971+ ) : null}
1972+ </p>
1973+ </div>
1974+ </div>
1975+ <h2>Submissions</h2>
1976+ <div class="box" style="overflow:auto">
1977+ <table class="tbl">
1978+ <thead>
1979+ <tr>
1980+ <th>When</th>
1981+ <th>URLs</th>
1982+ <th>IndexNow</th>
1983+ <th>Pages</th>
1984+ </tr>
1985+ </thead>
1986+ <tbody>
1987+ {s.submissions.map((r) => (
1988+ <tr>
1989+ <td class="small nowrap">
1990+ <Time at={r.at} />
1991+ </td>
1992+ <td>{r.urls}</td>
1993+ <td class="small">
1994+ <span class={`badge${r.status === 200 || r.status === 202 ? " green" : " red"}`}>{r.status ?? "unreachable"}</span> <span class="muted">{r.ms} ms</span>
1995+ {r.error ? <div class="mono muted">{r.error}</div> : null}
1996+ </td>
1997+ <td class="small mono" style="max-width:520px">
1998+ <details>
1999+ <summary style="overflow:hidden;text-overflow:ellipsis;white-space:nowrap">{r.paths.slice(0, 3).join(", ")}{r.urls > 3 ? ", …" : ""}</summary>
2000+ <div class="muted" style="white-space:pre-wrap">
2001+ {r.paths.join("\n")}
2002+ {r.urls > r.paths.length ? `\n… and ${r.urls - r.paths.length} more` : ""}
2003+ </div>
2004+ </details>
2005+ </td>
2006+ </tr>
2007+ ))}
2008+ </tbody>
2009+ </table>
2010+ {s.submissions.length === 0 ? <div class="empty">Nothing submitted yet. The first submission follows within five minutes of a deploy.</div> : null}
2011+ </div>
2012+ </>
2013+);
2014+
19272015 // ---------------------------------------------------------------- /:owner profiles
19282016
19292017 type OrgTab = "repositories" | "people" | "agents" | "settings";
addedapps/diffinery/src/web/indexnow.ts+323−0View file
@@ -1,0 +1,323 @@
1+/**
2+ * IndexNow (indexnow.org): tells search engines (Bing, Yandex, Naver, Seznam, Yep and the others that
3+ * share submissions) which public pages were added, changed or removed, so they recrawl those instead of
4+ * waiting to find out. Nobody has to run anything: the Worker's five-minute cron (app.ts `scheduled`) does it.
5+ *
6+ * - Site pages (the home page, the product, comparison and company pages, the docs, each one's .md
7+ * version, llms.txt and llms-full.txt) are built into the Worker, so they only change with a deploy.
8+ * The first cron run on a new version hashes each page's content, compares the hashes with the last
9+ * ones submitted and queues the pages that are new, changed or gone. A deploy that changes no page
10+ * submits nothing; later runs on the same version hash nothing, and a version older than the last one
11+ * checked (a run on an old isolate during a rollout) is ignored.
12+ * - Public repositories' pages are checked hourly: a repository pushed since it was last submitted is
13+ * queued. Only repositories that are public at that moment, with an owner who isn't suspended, are ever
14+ * submitted (the same ones the sitemap lists), and only in the same run that saw them public: a
15+ * private repository's page, or anything behind sign-in, is never sent.
16+ * - One POST to api.indexnow.org per run, at most MAX_URLS URLs (IndexNow takes 10,000). A refusal or a
17+ * failure to reach IndexNow keeps the pages queued and waits before trying again; it never touches a
18+ * deploy, which finished before the cron noticed it. Each submission is logged (console and
19+ * indexnow_submissions, shown on Settings → Search engines).
20+ *
21+ * The key is derived from MASTER_KEY and the instance's origin, so every instance has its own without
22+ * configuration, and is served at /<key>.txt. INDEXNOW=off turns all of it off; so does an origin that
23+ * isn't public (http, localhost).
24+ */
25+import { allPages, llmsFullTxt, llmsTxt, markdownUrl, pageMarkdown, pageUrl } from "../docs/pages.js";
26+import type { Env } from "../env.js";
27+import { hmacSha256 } from "../lib/crypto.js";
28+import type { Directory } from "../lib/directory.js";
29+import { earlyAccessOpen } from "../lib/early-access.js";
30+import { sha256 } from "../lib/util.js";
31+import { homeContent, homeMarkdown, sitePageMarkdown, sitePages } from "./site.js";
32+
33+export const INDEXNOW_ENDPOINT = "https://api.indexnow.org/indexnow";
34+/** URLs per submission: IndexNow's limit. */
35+export const MAX_URLS = 10_000;
36+/** Public repositories checked (and listed in the sitemap). */
37+export const MAX_REPOS = 5000;
38+/** How often public repositories are checked for pushes. */
39+export const REPO_INTERVAL_MS = 3600_000;
40+/** Submissions kept for Settings → Search engines. */
41+export const KEEP_SUBMISSIONS = 200;
42+/** How long to wait before trying again: after a 429, after another refusal, and when IndexNow couldn't be reached. */
43+export const RETRY_MS = { rateLimited: 3600_000, refused: 6 * 3600_000, unreachable: 15 * 60_000 };
44+const TIMEOUT_MS = 15_000;
45+/** Statements per D1 batch. */
46+const BATCH = 100;
47+
48+type IndexNowEnv = Pick<Env, "DB" | "PUBLIC_URL" | "MASTER_KEY" | "INDEXNOW" | "SIGNUP" | "CF_VERSION_METADATA">;
49+
50+const origin = (env: Pick<Env, "PUBLIC_URL">) => env.PUBLIC_URL.replace(/\/$/, "");
51+
52+/** Why IndexNow is off on this instance, or null when it is on. */
53+export function indexNowOff(env: Pick<Env, "PUBLIC_URL" | "MASTER_KEY" | "INDEXNOW">): string | null {
54+ if ((env.INDEXNOW ?? "").toLowerCase() === "off") return "INDEXNOW is off";
55+ let url: URL;
56+ try {
57+ url = new URL(env.PUBLIC_URL);
58+ } catch {
59+ return "PUBLIC_URL isn't a URL";
60+ }
61+ if (url.protocol !== "https:" || url.hostname === "localhost" || !url.hostname.includes(".") || /^[\d.]+$|:/.test(url.hostname)) return "PUBLIC_URL isn't a public https origin";
62+ if (!env.MASTER_KEY) return "MASTER_KEY isn't set";
63+ return null;
64+}
65+
66+const keys = new Map<string, Promise<string>>();
67+/** The instance's IndexNow key (32 hex characters), or null while IndexNow is off. */
68+export function indexNowKey(env: Pick<Env, "PUBLIC_URL" | "MASTER_KEY" | "INDEXNOW">): Promise<string> | null {
69+ if (indexNowOff(env)) return null;
70+ const id = `${env.MASTER_KEY} ${origin(env)}`;
71+ let k = keys.get(id);
72+ if (!k) {
73+ k = hmacSha256(env.MASTER_KEY!, `indexnow key for ${origin(env)}`).then((b) => [...b.slice(0, 16)].map((x) => x.toString(16).padStart(2, "0")).join(""));
74+ keys.set(id, k);
75+ }
76+ return k;
77+}
78+
79+/** Where the key file is served. */
80+export const keyLocation = (env: Pick<Env, "PUBLIC_URL">, key: string) => `${origin(env)}/${key}.txt`;
81+
82+/**
83+ * Every public page built into the Worker, by path, with the content that decides whether it changed:
84+ * the same pages as the sitemap (less public repositories), plus each one's Markdown version and llms.txt.
85+ */
86+export async function sitePageContents(env: Pick<Env, "PUBLIC_URL" | "SIGNUP">): Promise<Map<string, string>> {
87+ const o = origin(env);
88+ const pages = new Map<string, string>();
89+ pages.set("/", await homeContent(o));
90+ pages.set("/index.md", homeMarkdown(o));
91+ for (const p of sitePages()) {
92+ const md = sitePageMarkdown(p, o);
93+ pages.set(p.path, `${p.seoTitle ?? ""}\n${p.section}\n${md}`);
94+ pages.set(`${p.path}.md`, md);
95+ }
96+ // The early access form, while the instance is invite-only (as in the sitemap).
97+ if (earlyAccessOpen(env)) pages.set("/early-access", "early access");
98+ for (const p of allPages(o)) {
99+ const md = pageMarkdown(p, o);
100+ pages.set(pageUrl(p.slug), `${p.section}\n${md}`);
101+ pages.set(markdownUrl(p.slug), md);
102+ }
103+ pages.set("/llms.txt", llmsTxt(o));
104+ pages.set("/llms-full.txt", llmsFullTxt(o));
105+ return pages;
106+}
107+
108+const hashCache = new Map<string, Promise<{ hashes: Map<string, string>; fingerprint: string }>>();
109+/** Each site page's content hash and one fingerprint of them all. The pages are part of the deploy, so an isolate computes them once. */
110+export function siteHashes(env: Pick<Env, "PUBLIC_URL" | "SIGNUP">): Promise<{ hashes: Map<string, string>; fingerprint: string }> {
111+ const id = `${origin(env)} ${earlyAccessOpen(env)}`;
112+ let h = hashCache.get(id);
113+ if (!h) {
114+ h = (async () => {
115+ const hashes = new Map<string, string>();
116+ for (const [path, content] of await sitePageContents(env)) hashes.set(path, (await sha256(content)).slice(0, 32));
117+ const fingerprint = await sha256([...hashes].sort(([a], [b]) => (a < b ? -1 : 1)).map(([p, x]) => `${p} ${x}`).join("\n"));
118+ return { hashes, fingerprint };
119+ })();
120+ hashCache.set(id, h);
121+ h.catch(() => hashCache.delete(id));
122+ }
123+ return h;
124+}
125+
126+// ---------------------------------------------------------------- state
127+
128+async function getState(db: D1Database, k: string): Promise<string | null> {
129+ return (await db.prepare("SELECT v FROM indexnow_state WHERE k = ?").bind(k).first<{ v: string }>())?.v ?? null;
130+}
131+const setState = (db: D1Database, k: string, v: string) => db.prepare("INSERT INTO indexnow_state (k, v) VALUES (?, ?) ON CONFLICT(k) DO UPDATE SET v = excluded.v").bind(k, v);
132+
133+async function runBatches(db: D1Database, stmts: D1PreparedStatement[]): Promise<void> {
134+ for (let i = 0; i < stmts.length; i += BATCH) await db.batch(stmts.slice(i, i + BATCH));
135+}
136+
137+type Row = { path: string; kind: string; hash: string; sent_hash: string | null };
138+
139+/**
140+ * Queues the site pages a deploy added, changed or removed. Runs only when this version's content
141+ * fingerprint differs from the last one checked, and never for a version older than that one's.
142+ * Returns how many pages it queued.
143+ */
144+export async function checkSite(env: IndexNowEnv): Promise<number> {
145+ const db = env.DB;
146+ const version = env.CF_VERSION_METADATA;
147+ const [lastFingerprint, lastVersionAt, lastVersionId] = await Promise.all([getState(db, "site_fingerprint"), getState(db, "site_version_at"), getState(db, "site_version_id")]);
148+ // This version was checked already, or a newer one was: nothing to hash.
149+ if (version?.id && version.id === lastVersionId) return 0;
150+ if (version?.timestamp && lastVersionAt && version.timestamp < lastVersionAt) return 0;
151+ const { hashes, fingerprint } = await siteHashes(env);
152+ const seen = version?.id && version.timestamp ? [setState(db, "site_version_id", version.id), setState(db, "site_version_at", version.timestamp)] : [];
153+ if (fingerprint === lastFingerprint) {
154+ if (seen.length) await db.batch(seen);
155+ return 0;
156+ }
157+ const rows = (await db.prepare("SELECT path, kind, hash, sent_hash FROM indexnow_urls WHERE kind = 'site'").all<Row>()).results;
158+ const known = new Map(rows.map((r) => [r.path, r]));
159+ const stmts: D1PreparedStatement[] = [];
160+ for (const [path, hash] of hashes) {
161+ if (known.get(path)?.hash === hash) continue;
162+ stmts.push(db.prepare("INSERT INTO indexnow_urls (path, kind, hash) VALUES (?, 'site', ?) ON CONFLICT(path) DO UPDATE SET kind = 'site', hash = excluded.hash").bind(path, hash));
163+ }
164+ for (const r of rows) {
165+ if (hashes.has(r.path) || r.hash === "") continue;
166+ // Gone: a page that was submitted is submitted once more, so search engines see it is gone; one that never was is forgotten.
167+ stmts.push(r.sent_hash === null ? db.prepare("DELETE FROM indexnow_urls WHERE path = ?").bind(r.path) : db.prepare("UPDATE indexnow_urls SET hash = '' WHERE path = ?").bind(r.path));
168+ }
169+ const queued = stmts.length;
170+ stmts.push(setState(db, "site_fingerprint", fingerprint), ...seen);
171+ await runBatches(db, stmts);
172+ return queued;
173+}
174+
175+/**
176+ * Queues public repositories pushed since they were last submitted, and forgets repositories that are no
177+ * longer public (they are never submitted again). Returns the paths of every public repository now
178+ * queued: only these may be submitted in this run.
179+ */
180+export async function checkRepos(env: IndexNowEnv, dir: Pick<Directory, "publicRepos">): Promise<Set<string>> {
181+ const db = env.DB;
182+ const repos = await dir.publicRepos(MAX_REPOS);
183+ const current = new Map(repos.map((r) => [`/${encodeURIComponent(r.owner_login)}/${encodeURIComponent(r.name)}`, r.pushed_at ?? r.updated_at]));
184+ const rows = (await db.prepare("SELECT path, kind, hash, sent_hash FROM indexnow_urls WHERE kind = 'repo'").all<Row>()).results;
185+ const known = new Map(rows.map((r) => [r.path, r]));
186+ const stmts: D1PreparedStatement[] = [];
187+ for (const r of rows) if (!current.has(r.path)) stmts.push(db.prepare("DELETE FROM indexnow_urls WHERE path = ? AND kind = 'repo'").bind(r.path));
188+ const queued = new Set<string>();
189+ for (const [path, at] of current) {
190+ const row = known.get(path);
191+ if (row?.hash !== at) stmts.push(db.prepare("INSERT INTO indexnow_urls (path, kind, hash) VALUES (?, 'repo', ?) ON CONFLICT(path) DO UPDATE SET hash = excluded.hash").bind(path, at));
192+ if (row?.sent_hash !== at) queued.add(path);
193+ }
194+ stmts.push(setState(db, "repos_checked_at", new Date().toISOString()));
195+ await runBatches(db, stmts);
196+ return queued;
197+}
198+
199+// ---------------------------------------------------------------- submitting
200+
201+export type Submission = { at: string; urls: number; status: number | null; ms: number; error: string | null; paths: string[] };
202+
203+/** POSTs `paths` to IndexNow. Never throws. */
204+export async function submit(env: Pick<Env, "PUBLIC_URL">, key: string, paths: string[], fetcher: typeof fetch = fetch): Promise<Submission> {
205+ const o = origin(env);
206+ const at = new Date().toISOString();
207+ const started = Date.now();
208+ const body = JSON.stringify({ host: new URL(o).host, key, keyLocation: keyLocation(env, key), urlList: paths.map((p) => `${o}${p}`) });
209+ let status: number | null = null;
210+ let error: string | null = null;
211+ try {
212+ const res = await fetcher(INDEXNOW_ENDPOINT, { method: "POST", headers: { "content-type": "application/json; charset=utf-8" }, body, signal: AbortSignal.timeout(TIMEOUT_MS) });
213+ status = res.status;
214+ const text = (await res.text().catch(() => "")).replace(/\s+/g, " ").trim();
215+ if (status !== 200 && status !== 202) error = text.slice(0, 500) || res.statusText || `HTTP ${status}`;
216+ } catch (e) {
217+ error = (e instanceof Error ? `${e.name}: ${e.message}` : String(e)).slice(0, 500);
218+ }
219+ return { at, urls: paths.length, status, ms: Date.now() - started, error, paths };
220+}
221+
222+const accepted = (s: Submission) => s.status === 200 || s.status === 202;
223+
224+/**
225+ * The cron's part: queue what a deploy changed (and, hourly, what was pushed to public repositories),
226+ * then submit the queue. Never throws; a failure is logged and tried again later.
227+ */
228+export async function indexNowCron(env: IndexNowEnv, dir: Pick<Directory, "publicRepos">, fetcher: typeof fetch = fetch): Promise<Submission | null> {
229+ const key = indexNowKey(env);
230+ if (!key) return null;
231+ const db = env.DB;
232+ try {
233+ const nowMs = Date.now();
234+ const [retryAt, reposAt] = await Promise.all([getState(db, "retry_at"), getState(db, "repos_checked_at")]);
235+ const waiting = !!retryAt && Date.parse(retryAt) > nowMs;
236+ const siteQueued = await checkSite(env);
237+ if (waiting) return null;
238+ // Repositories once an hour, and only when IndexNow isn't asking us to wait.
239+ const repos = !reposAt || nowMs - Date.parse(reposAt) >= REPO_INTERVAL_MS ? await checkRepos(env, dir) : new Set<string>();
240+ // Pages wait in the queue only after a refusal (retry_at) or when one submission couldn't take them all.
241+ if (!siteQueued && !repos.size && !retryAt) return null;
242+
243+ const site = (await db.prepare("SELECT path, kind, hash, sent_hash FROM indexnow_urls WHERE kind = 'site' AND (sent_hash IS NULL OR sent_hash != hash)").all<Row>()).results;
244+ const repoRows = repos.size ? (await db.prepare("SELECT path, kind, hash, sent_hash FROM indexnow_urls WHERE kind = 'repo' AND (sent_hash IS NULL OR sent_hash != hash)").all<Row>()).results.filter((r) => repos.has(r.path)) : [];
245+ const all = [...site, ...repoRows];
246+ if (!all.length) {
247+ if (retryAt) await db.prepare("DELETE FROM indexnow_state WHERE k = 'retry_at'").run();
248+ return null;
249+ }
250+ const batch = all.slice(0, MAX_URLS);
251+ const s = await submit(env, await key, batch.map((r) => r.path), fetcher);
252+ console.log(`indexnow: ${s.urls} URLs, ${s.status ?? "unreachable"} in ${s.ms} ms${s.error ? `: ${s.error}` : ""}`);
253+ const stmts: D1PreparedStatement[] = [
254+ db.prepare("INSERT INTO indexnow_submissions (at, urls, status, ms, error, paths) VALUES (?, ?, ?, ?, ?, ?)").bind(s.at, s.urls, s.status, s.ms, s.error, JSON.stringify(s.paths.slice(0, 50))),
255+ db.prepare("DELETE FROM indexnow_submissions WHERE id <= (SELECT id FROM indexnow_submissions ORDER BY id DESC LIMIT 1 OFFSET ?)").bind(KEEP_SUBMISSIONS),
256+ ];
257+ if (accepted(s)) {
258+ for (const r of batch) {
259+ stmts.push(r.hash === "" ? db.prepare("DELETE FROM indexnow_urls WHERE path = ? AND hash = ''").bind(r.path) : db.prepare("UPDATE indexnow_urls SET sent_hash = ?, sent_at = ? WHERE path = ?").bind(r.hash, s.at, r.path));
260+ }
261+ // More than one submission holds: the rest go in the next run (site pages; repositories after the next check).
262+ stmts.push(all.length > batch.length ? setState(db, "retry_at", s.at) : db.prepare("DELETE FROM indexnow_state WHERE k = 'retry_at'"));
263+ } else {
264+ const wait = s.status === 429 ? RETRY_MS.rateLimited : s.status === null || s.status >= 500 ? RETRY_MS.unreachable : RETRY_MS.refused;
265+ stmts.push(setState(db, "retry_at", new Date(nowMs + wait).toISOString()));
266+ }
267+ await runBatches(db, stmts);
268+ return s;
269+ } catch (e) {
270+ console.error("indexnow failed", e);
271+ return null;
272+ }
273+}
274+
275+// ---------------------------------------------------------------- what administrators and /meta see
276+
277+export type IndexNowStatus = {
278+ off: string | null;
279+ keyLocation: string | null;
280+ site: number;
281+ repos: number;
282+ queued: number;
283+ retryAt: string | null;
284+ reposCheckedAt: string | null;
285+ submissions: Submission[];
286+};
287+
288+export async function indexNowStatus(env: IndexNowEnv, limit = 50): Promise<IndexNowStatus> {
289+ const off = indexNowOff(env);
290+ const key = indexNowKey(env);
291+ const [counts, retryAt, reposCheckedAt, subs] = await Promise.all([
292+ env.DB.prepare("SELECT kind, COUNT(*) AS n, SUM(CASE WHEN sent_hash IS NULL OR sent_hash != hash THEN 1 ELSE 0 END) AS queued FROM indexnow_urls GROUP BY kind").all<{ kind: string; n: number; queued: number }>(),
293+ getState(env.DB, "retry_at"),
294+ getState(env.DB, "repos_checked_at"),
295+ lastSubmissions(env.DB, limit),
296+ ]);
297+ const by = new Map(counts.results.map((r) => [r.kind, r]));
298+ return {
299+ off,
300+ keyLocation: key ? keyLocation(env, await key) : null,
301+ site: by.get("site")?.n ?? 0,
302+ repos: by.get("repo")?.n ?? 0,
303+ queued: counts.results.reduce((n, r) => n + (r.queued ?? 0), 0),
304+ retryAt,
305+ reposCheckedAt,
306+ submissions: subs,
307+ };
308+}
309+
310+export async function lastSubmissions(db: D1Database, limit: number): Promise<Submission[]> {
311+ const rows = (await db.prepare("SELECT at, urls, status, ms, error, paths FROM indexnow_submissions ORDER BY id DESC LIMIT ?").bind(limit).all<Omit<Submission, "paths"> & { paths: string }>()).results;
312+ return rows.map((r) => ({ ...r, paths: JSON.parse(r.paths) as string[] }));
313+}
314+
315+/** For GET /api/v3/meta: whether IndexNow is on, its key file, and how the last submission went (no URLs). */
316+export async function indexNowMeta(env: IndexNowEnv): Promise<{ enabled: boolean; key_location: string | null; last_submission: { at: string; urls: number; status: number | null } | null }> {
317+ const key = indexNowKey(env);
318+ if (!key) return { enabled: false, key_location: null, last_submission: null };
319+ const last = await env.DB.prepare("SELECT at, urls, status FROM indexnow_submissions ORDER BY id DESC LIMIT 1")
320+ .first<{ at: string; urls: number; status: number | null }>()
321+ .catch(() => null);
322+ return { enabled: true, key_location: keyLocation(env, await key), last_submission: last ?? null };
323+}
modifiedapps/diffinery/src/web/seo.tsx+7−0View file
@@ -14,6 +14,7 @@
1414 import { ICON_FILES } from "./icons.generated.js";
1515 import { docsSitemapPaths } from "../docs/web.js";
1616 import { siteSitemapEntries } from "./site.js";
17+import { indexNowKey } from "./indexnow.js";
1718
1819 export const SITE = {
1920 name: "Diffinery",
@@ -89,6 +90,12 @@
8990 ];
9091 return c.text(lines.join("\n"), 200, { "cache-control": "public, max-age=86400" });
9192 });
93+ // The IndexNow key file (web/indexnow.ts): /<key>.txt holds the key, which proves submissions come from this site.
94+ web.get("/:file{[0-9a-f]+\\.txt}", async (c) => {
95+ const key = await indexNowKey(c.env);
96+ if (!key || c.req.param("file") !== `${key}.txt`) return c.notFound();
97+ return c.text(key, 200, { "cache-control": "public, max-age=3600" });
98+ });
9299 web.get("/sitemap.xml", async (c) => {
93100 const origin = c.env.PUBLIC_URL.replace(/\/$/, "");
94101 const repos = await c.get("dir").publicRepos(5000);
modifiedapps/diffinery/src/web/site.tsx+5−0View file
@@ -195,6 +195,11 @@
195195 return absolute(lines.join("\n"), o);
196196 }
197197
198+/** Everything the home page says, for IndexNow (web/indexnow.ts): its Markdown and the landing page's own copy. */
199+export async function homeContent(o: string): Promise<string> {
200+ return `${homeMarkdown(o)}\n${await (<Landing early={false} />).toString()}`;
201+}
202+
198203 /** schema.org ids for the site's entities, shared by every page's graph. */
199204 const ids = (o: string) => ({ org: `${o}/#organization`, site: `${o}/#website`, app: `${o}/#software` });
200205
modifiedapps/diffinery/test/e2e/e2e.test.mjs+6−0View file
@@ -2814,6 +2814,12 @@
28142814 const errors = await ok("GET", "/diffinery/admin/errors?per_page=5");
28152815 assert.ok(Array.isArray(errors) && errors.length <= 5, JSON.stringify(errors));
28162816 assert.equal((await api("GET", "/diffinery/admin/errors/not-a-request-id")).status, 404);
2817+ // IndexNow is off on a local origin; its page and API still answer for site admins.
2818+ const indexnow = await ok("GET", "/diffinery/admin/indexnow");
2819+ assert.equal(indexnow.enabled, false, JSON.stringify(indexnow));
2820+ assert.match(indexnow.off_reason, /public https/);
2821+ assert.match(await (await fetch(`${base}/settings/admin/search-engines`)).text(), /Search engines[\s\S]*PUBLIC_URL isn/);
2822+ assert.equal((await ok("GET", "/meta")).indexnow.enabled, false);
28172823 // A push without the schedule removes it.
28182824 const src = join(work, "schedules");
28192825 rmSync(join(src, ".github/workflows/tick.yml"));
addedapps/diffinery/test/unit/indexnow.test.ts+237−0View file
@@ -1,0 +1,237 @@
1+/**
2+ * IndexNow (web/indexnow.ts): the key file, which pages are covered, and the cron that submits only
3+ * what changed, never a private repository, and never throws.
4+ * Needs src/docs/content.generated.ts (`pnpm build` or `pnpm build:docs`).
5+ */
6+import { beforeEach, describe, expect, it } from "vitest";
7+import { app } from "../../src/app.js";
8+import { web } from "../../src/web/routes.js";
9+import type { Env } from "../../src/env.js";
10+import type { Repo } from "../../src/lib/directory.js";
11+import { docsSitemapPaths } from "../../src/docs/web.js";
12+import { siteSitemapEntries } from "../../src/web/site.js";
13+import { indexNowCron, indexNowKey, indexNowOff, indexNowStatus, MAX_URLS, RETRY_MS, sitePageContents } from "../../src/web/indexnow.js";
14+import { testD1 } from "./stubs/d1.js";
15+
16+app.route("/", web);
17+
18+const ORIGIN = "https://diffinery.example";
19+const MASTER_KEY = btoa("k".repeat(32));
20+const ctx = { waitUntil() {}, passThroughOnException() {}, props: {} } as unknown as ExecutionContext;
21+
22+type Sent = { host: string; key: string; keyLocation: string; urlList: string[] };
23+
24+function setup(o: { status?: number | (() => number); version?: string } = {}) {
25+ const db = testD1();
26+ const env = { PUBLIC_URL: ORIGIN, MASTER_KEY, DB: db, CF_VERSION_METADATA: o.version ? { id: "v", tag: "", timestamp: o.version } : undefined } as unknown as Env;
27+ const sent: Sent[] = [];
28+ let status = o.status ?? 200;
29+ const fetcher = (async (url: string, init: RequestInit) => {
30+ expect(url).toBe("https://api.indexnow.org/indexnow");
31+ expect(new Headers(init.headers).get("content-type")).toBe("application/json; charset=utf-8");
32+ sent.push(JSON.parse(init.body as string) as Sent);
33+ const s = typeof status === "function" ? status() : status;
34+ return new Response(s === 200 ? "" : "nope", { status: s });
35+ }) as unknown as typeof fetch;
36+ let repos: Array<Partial<Repo>> = [];
37+ const dir = { publicRepos: async () => repos as Repo[] };
38+ return {
39+ env,
40+ db,
41+ sent,
42+ run: () => indexNowCron(env as never, dir, fetcher),
43+ setRepos: (r: Array<Partial<Repo>>) => (repos = r),
44+ /** A new version, as a deploy makes (its content is the same build, so tests change what was last submitted instead). */
45+ deploy: (timestamp: string) => ((env as { CF_VERSION_METADATA?: WorkerVersionMetadata }).CF_VERSION_METADATA = { id: `v-${timestamp}`, tag: "", timestamp }),
46+ setStatus: (s: number) => (status = s),
47+ state: (k: string) => db.sqlite.prepare("SELECT v FROM indexnow_state WHERE k = ?").get(k) as { v: string } | undefined,
48+ /** Moves time forward for the cron: every stored time goes back by `ms`. */
49+ age: (ms: number) => {
50+ for (const k of ["retry_at", "repos_checked_at"]) {
51+ const r = db.sqlite.prepare("SELECT v FROM indexnow_state WHERE k = ?").get(k) as { v: string } | undefined;
52+ if (r) db.sqlite.prepare("UPDATE indexnow_state SET v = ? WHERE k = ?").run(new Date(Date.parse(r.v) - ms).toISOString(), k);
53+ }
54+ },
55+ };
56+}
57+
58+const paths = (s: Sent) => s.urlList.map((u) => u.slice(ORIGIN.length));
59+
60+describe("IndexNow", () => {
61+ it("is on only for a public https origin with a master key, unless INDEXNOW=off", () => {
62+ expect(indexNowOff({ PUBLIC_URL: ORIGIN, MASTER_KEY })).toBeNull();
63+ expect(indexNowOff({ PUBLIC_URL: ORIGIN, MASTER_KEY, INDEXNOW: "off" })).toMatch(/off/);
64+ expect(indexNowOff({ PUBLIC_URL: "http://localhost:8787", MASTER_KEY })).toMatch(/public/);
65+ expect(indexNowOff({ PUBLIC_URL: "https://localhost", MASTER_KEY })).toMatch(/public/);
66+ expect(indexNowOff({ PUBLIC_URL: "https://10.0.0.1", MASTER_KEY })).toMatch(/public/);
67+ expect(indexNowOff({ PUBLIC_URL: ORIGIN })).toMatch(/MASTER_KEY/);
68+ expect(indexNowKey({ PUBLIC_URL: "http://localhost:8787", MASTER_KEY })).toBeNull();
69+ });
70+
71+ it("serves the key file at the site root, and nothing at other names", async () => {
72+ const env = { PUBLIC_URL: ORIGIN, MASTER_KEY, DB: testD1() } as unknown as Env;
73+ const key = await indexNowKey(env)!;
74+ expect(key).toMatch(/^[0-9a-f]{32}$/);
75+ // Stable for the instance, different for another origin or master key.
76+ expect(await indexNowKey(env)!).toBe(key);
77+ expect(await indexNowKey({ PUBLIC_URL: "https://other.example", MASTER_KEY })!).not.toBe(key);
78+ expect(await indexNowKey({ PUBLIC_URL: ORIGIN, MASTER_KEY: btoa("j".repeat(32)) })!).not.toBe(key);
79+ const res = await app.fetch(new Request(`${ORIGIN}/${key}.txt`), env, ctx);
80+ expect(res.status).toBe(200);
81+ expect(res.headers.get("content-type")).toMatch(/^text\/plain/);
82+ expect(await res.text()).toBe(key);
83+ expect((await app.fetch(new Request(`${ORIGIN}/${"0".repeat(32)}.txt`), env, ctx)).status).toBe(404);
84+ const off = { ...env, INDEXNOW: "off" } as Env;
85+ expect((await app.fetch(new Request(`${ORIGIN}/${key}.txt`), off, ctx)).status).toBe(404);
86+ });
87+
88+ it("covers every page in the sitemap and each one's Markdown version, and nothing behind sign-in", async () => {
89+ const pages = await sitePageContents({ PUBLIC_URL: ORIGIN });
90+ const want = [...siteSitemapEntries().map((e) => e.path), ...docsSitemapPaths(), "/early-access"];
91+ for (const p of want) expect(pages.has(p), p).toBe(true);
92+ for (const p of siteSitemapEntries().map((e) => e.path)) expect(pages.has(p === "/" ? "/index.md" : `${p}.md`), `${p}.md`).toBe(true);
93+ for (const p of docsSitemapPaths()) expect(pages.has(p === "/docs" ? "/docs/index.md" : `${p}.md`), `${p}.md`).toBe(true);
94+ for (const p of ["/llms.txt", "/llms-full.txt", "/compare/github", "/docs/quickstart", "/docs/quickstart.md"]) expect(pages.has(p), p).toBe(true);
95+ for (const p of pages.keys()) expect(p).not.toMatch(/^\/(settings|login|signin|auth|admin|new|notifications|invite|api|-)(\/|$)/);
96+ // Every page answers 200 to a visitor.
97+ const env = { PUBLIC_URL: ORIGIN, DB: testD1() } as unknown as Env;
98+ for (const p of ["/", "/index.md", "/compare/github.md", "/docs/quickstart.md", "/llms.txt"]) expect((await app.fetch(new Request(`${ORIGIN}${p}`), env, ctx)).status, p).toBe(200);
99+ // Open sign-up: no early access form.
100+ expect((await sitePageContents({ PUBLIC_URL: ORIGIN, SIGNUP: "open" })).has("/early-access")).toBe(false);
101+ });
102+
103+ describe("cron", () => {
104+ let t: ReturnType<typeof setup>;
105+ beforeEach(() => {
106+ t = setup({ version: "2026-10-22T10:00:00.000Z" });
107+ });
108+
109+ it("submits every page once, then only what a deploy changed or removed", async () => {
110+ const all = await sitePageContents({ PUBLIC_URL: ORIGIN });
111+ const s = await t.run();
112+ expect(s).toMatchObject({ status: 200, error: null, urls: all.size });
113+ expect(t.sent).toHaveLength(1);
114+ const key = await indexNowKey(t.env)!;
115+ expect(t.sent[0]).toMatchObject({ host: "diffinery.example", key, keyLocation: `${ORIGIN}/${key}.txt` });
116+ expect(new Set(paths(t.sent[0]!))).toEqual(new Set(all.keys()));
117+ expect(t.sent[0]!.urlList.every((u) => u.startsWith(`${ORIGIN}/`))).toBe(true);
118+
119+ // Nothing changed: nothing sent.
120+ expect(await t.run()).toBeNull();
121+ expect(t.sent).toHaveLength(1);
122+
123+ // A deploy that changed one page and removed another (as if they had held other content before).
124+ t.db.sqlite.prepare("UPDATE indexnow_urls SET hash = 'old', sent_hash = 'old' WHERE path = '/docs/quickstart'").run();
125+ t.db.sqlite.prepare("INSERT INTO indexnow_urls (path, kind, hash, sent_hash) VALUES ('/docs/removed', 'site', 'x', 'x'), ('/docs/never-sent', 'site', 'y', NULL)").run();
126+ t.db.sqlite.prepare("UPDATE indexnow_state SET v = 'older' WHERE k = 'site_fingerprint'").run();
127+ // The same version doesn't look again.
128+ expect(await t.run()).toBeNull();
129+ t.deploy("2026-10-22T11:00:00.000Z");
130+ expect(await t.run()).toMatchObject({ status: 200, urls: 2 });
131+ expect(paths(t.sent[1]!).sort()).toEqual(["/docs/quickstart", "/docs/removed"]);
132+ // A page gone and submitted is forgotten; one never submitted was forgotten without a submission.
133+ expect(t.db.sqlite.prepare("SELECT path FROM indexnow_urls WHERE path IN ('/docs/removed', '/docs/never-sent')").all()).toEqual([]);
134+ expect(await t.run()).toBeNull();
135+ expect(t.sent).toHaveLength(2);
136+ });
137+
138+ it("ignores an older version that runs during a rollout", async () => {
139+ await t.run();
140+ t.db.sqlite.prepare("UPDATE indexnow_urls SET hash = 'old', sent_hash = 'old' WHERE path = '/features'").run();
141+ t.db.sqlite.prepare("UPDATE indexnow_state SET v = 'other' WHERE k = 'site_fingerprint'").run();
142+ const old = setup({ version: "2026-10-22T09:00:00.000Z" });
143+ (old.env as { DB: D1Database }).DB = t.env.DB;
144+ const fetcher = (async () => new Response("", { status: 200 })) as unknown as typeof fetch;
145+ expect(await indexNowCron(old.env as never, { publicRepos: async () => [] }, fetcher)).toBeNull();
146+ expect(t.state("site_fingerprint")?.v).toBe("other");
147+ // A newer version does check.
148+ t.deploy("2026-10-22T11:00:00.000Z");
149+ expect(await t.run()).toMatchObject({ urls: 1 });
150+ expect(paths(t.sent[1]!)).toEqual(["/features"]);
151+ });
152+
153+ it("submits public repositories when pushed, hourly, and never a private one", async () => {
154+ t.setRepos([
155+ { owner_login: "acme", name: "app", pushed_at: "2026-10-22T09:00:00Z", updated_at: "2026-10-01T00:00:00Z" },
156+ { owner_login: "acme", name: "lib", pushed_at: null, updated_at: "2026-10-02T00:00:00Z" },
157+ ]);
158+ await t.run();
159+ expect(paths(t.sent[0]!)).toEqual(expect.arrayContaining(["/acme/app", "/acme/lib"]));
160+ // Pushed again within the hour: waits for the hourly check.
161+ t.setRepos([
162+ { owner_login: "acme", name: "app", pushed_at: "2026-10-22T09:30:00Z" },
163+ { owner_login: "acme", name: "lib", pushed_at: null, updated_at: "2026-10-02T00:00:00Z" },
164+ ]);
165+ expect(await t.run()).toBeNull();
166+ t.age(3600_000);
167+ expect(await t.run()).toMatchObject({ urls: 1 });
168+ expect(paths(t.sent[1]!)).toEqual(["/acme/app"]);
169+ // acme/app went private (publicRepos no longer lists it) after a push: never submitted, and forgotten.
170+ t.db.sqlite.prepare("UPDATE indexnow_urls SET hash = 'pushed-again' WHERE path = '/acme/app'").run();
171+ t.setRepos([{ owner_login: "acme", name: "lib", pushed_at: null, updated_at: "2026-10-02T00:00:00Z" }]);
172+ t.age(3600_000);
173+ expect(await t.run()).toBeNull();
174+ expect(t.sent).toHaveLength(2);
175+ expect(t.db.sqlite.prepare("SELECT path FROM indexnow_urls WHERE kind = 'repo'").all()).toEqual([{ path: "/acme/lib" }]);
176+ // A queued repository is only submitted in a run that saw it public.
177+ t.db.sqlite.prepare("UPDATE indexnow_urls SET sent_hash = NULL WHERE path = '/acme/lib'").run();
178+ t.db.sqlite.prepare("UPDATE indexnow_urls SET sent_hash = 'old' WHERE path = '/features'").run();
179+ t.db.sqlite.prepare("INSERT INTO indexnow_state (k, v) VALUES ('retry_at', '2000-01-01T00:00:00Z')").run();
180+ expect(await t.run()).toMatchObject({ urls: 1 });
181+ expect(paths(t.sent[2]!)).toEqual(["/features"]);
182+ });
183+
184+ it("keeps pages queued after a refusal and waits before trying again", async () => {
185+ t.setStatus(429);
186+ const s = await t.run();
187+ expect(s).toMatchObject({ status: 429, error: "nope" });
188+ const retry = Date.parse(t.state("retry_at")!.v);
189+ expect(retry - Date.now()).toBeGreaterThan(RETRY_MS.rateLimited - 60_000);
190+ // Waiting: nothing sent.
191+ expect(await t.run()).toBeNull();
192+ expect(t.sent).toHaveLength(1);
193+ // Once the wait is over, the same pages go again.
194+ t.setStatus(202);
195+ t.age(RETRY_MS.rateLimited);
196+ expect(await t.run()).toMatchObject({ status: 202, urls: s!.urls });
197+ expect(t.state("retry_at")).toBeUndefined();
198+ expect(await t.run()).toBeNull();
199+ const st = await indexNowStatus(t.env as never);
200+ expect(st).toMatchObject({ off: null, queued: 0, retryAt: null });
201+ expect(st.submissions.map((x) => x.status)).toEqual([202, 429]);
202+ });
203+
204+ it("never throws when IndexNow can't be reached, and tries again later", async () => {
205+ const fetcher = (async () => {
206+ throw new TypeError("network down");
207+ }) as unknown as typeof fetch;
208+ const s = await indexNowCron(t.env as never, { publicRepos: async () => [] }, fetcher);
209+ expect(s).toMatchObject({ status: null, error: "TypeError: network down" });
210+ expect(Date.parse(t.state("retry_at")!.v)).toBeGreaterThan(Date.now());
211+ // A broken database is logged, not thrown.
212+ const broken = { ...t.env, DB: { prepare: () => { throw new Error("D1 down"); } } };
213+ expect(await indexNowCron(broken as never, { publicRepos: async () => [] }, fetcher)).toBeNull();
214+ });
215+
216+ it("sends at most MAX_URLS in one submission, and the rest in the next run", async () => {
217+ const many = Array.from({ length: MAX_URLS }, (_, i) => ({ owner_login: "o", name: `r${i}`, pushed_at: "2026-10-22T09:00:00Z" }));
218+ t.setRepos(many);
219+ const site = (await sitePageContents({ PUBLIC_URL: ORIGIN })).size;
220+ // checkRepos reads at most MAX_REPOS; the fake hands back them all, which stands in for a large queue.
221+ expect(await t.run()).toMatchObject({ urls: MAX_URLS });
222+ expect(await t.run()).toBeNull(); // repositories wait for the next hourly check
223+ t.age(3600_000);
224+ expect(await t.run()).toMatchObject({ urls: site });
225+ expect(await t.run()).toBeNull();
226+ });
227+ });
228+
229+ it("/api/v3/meta says whether IndexNow is on and how the last submission went", async () => {
230+ const t = setup();
231+ await t.run();
232+ const res = await app.fetch(new Request(`${ORIGIN}/api/v3/meta`), t.env, ctx);
233+ const meta = (await res.json()) as { indexnow: { enabled: boolean; key_location: string; last_submission: { urls: number; status: number } } };
234+ expect(meta.indexnow).toMatchObject({ enabled: true, key_location: `${ORIGIN}/${await indexNowKey(t.env)!}.txt`, last_submission: { status: 200 } });
235+ expect(JSON.stringify(meta)).not.toContain("/docs/");
236+ });
237+});
modifiedapps/diffinery/wrangler.jsonc+3−0View file
@@ -12,6 +12,8 @@
1212 // route runs the Worker in front of them as long as the record is proxied (orange cloud).
1313 "routes": [{ "pattern": "diffinery.com/*", "zone_name": "diffinery.com" }],
1414 "observability": { "enabled": true, "head_sampling_rate": 1 },
15+ // The deployed version's id and time: IndexNow checks the site's pages once per new version (src/web/indexnow.ts).
16+ "version_metadata": { "binding": "CF_VERSION_METADATA" },
1517 "upload_source_maps": true,
1618 "limits": { "cpu_ms": 300000 },
1719
@@ -131,6 +133,7 @@
131133 // Containers do not run under wrangler dev: an import agent started by hand (or by the e2e test) claims Git transfers.
132134 "IMPORT_GIT_AGENT": "external"
133135 },
136+ "version_metadata": { "binding": "CF_VERSION_METADATA" },
134137 "d1_databases": [{ "binding": "DB", "database_name": "diffinery-dev", "database_id": "00000000-0000-0000-0000-000000000000", "migrations_dir": "migrations" }],
135138 "r2_buckets": [{ "binding": "BUCKET", "bucket_name": "diffinery-data-dev" }],
136139 "queues": {