[perber]

Last step of the SEO series for #1292 (after #1795, which added the shell Vary seam and the locked route reads). Step "rendered body for search-engine crawlers" from ADR-0018.

Search engines that don't run JavaScript (and Google's first pass) see an empty #root. Public wikis now put the page into #root for known crawlers; the SPA's createRoot replaces it, so browsers are unaffected.

What's in it

• seo/crawler_body.go: isSearchEngineCrawler (case-insensitive substrings googlebot, bingbot, duckduckbot, yandex, baiduspider, applebot, google-inspectiontool) and GetCrawlerBodyUseCase: <article><h1>escaped title</h1> + render.RenderHTML (ADR-0020 renderer and limits, on main since feat(render): render sanitized page HTML for crawlers #1789/fix(render): bound render cost and fail closed outside the policy #1792) + <nav><ul> of escaped child links.
• Contributor behind PublicOnly.Contributor, after the page meta contributor. Every public shell gets Vary: User-Agent and no-cache.
• Cache: true LRU of rendered bodies, 16 MB, keyed by SHA-256 of base path, route path and content; a hit is reused only while every wikilink in it still resolves the same way; failed renders are cached too.
• At most 2 renders at a time; a request beyond that gets the plain shell instead of waiting; a client that is gone when it gets a slot gets no render.
• Errors and ErrTooComplex → plain shell, Warn once per page version / once per process, never page content in the log.
• No body for /p/ permalinks, the root, unknown pages or app screens; a crawler that asks for text/markdown gets the Markdown.
• Reads through the pages slice only: ListChildRoutesUseCase (new TreeService.ChildRoutes, locked) and ResolveWikiLinkUseCase (exported links.ResolveWikiLinkTarget over the locked route reads).
• ADR-0018 "Later changes" (incl. what scoped reads must do once hidden pages exist), ADR-0020 → Accepted, source map.
• e2e seo-crawler-body.spec.ts: a Googlebot UA gets the article in #root, the SPA replaces it (content once, at most one H1); a browser UA gets an empty #root.

Behavior-Change: in public wikis, requests from search-engine crawlers get the rendered page in #root; every public shell response carries Vary: User-Agent.

Checks

• make test, make lint, deadcode, make verify, make cross, make verify-commits green; -race on seo, pages, tree, links and the real-router crawler tests (fresh frontend build); e2e crawler + meta specs pass.
• Tester (incl. hostile titles/content/links, cache following edits/moves/renames, concurrency), architecture, code and security review: nothing blocking after two fix rounds (race fix moved into refactor: locked route reads for wikilinks and Vary for shell contributors #1795, LRU, client-gone check, reads via pages use cases, ADR updates).

For your review

• Size: ~514 production lines, slightly above the 400–500 band; the remaining parts (child routes, wikilink use case, LRU, context check) came in through review and splitting them further would leave unused API in a PR. Total diff is large mainly because of tests.
• A crawler request reads the page twice (meta contributor + crawler body); sharing it would need request-scoped state across contributors, which the seam doesn't have.
• yandex substring breadth, the cap of 2 renders and the 16 MB cache are not measured on a Raspberry Pi yet.
• Same content as the SPA, so not cloaking; Vary: User-Agent disables shared caching of the shell (it is no-cache anyway).
• Manual check after merge: curl -A 'Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)' against a deployed public wiki (proxy/CDN Vary handling), then Search Console URL Inspection (live test uses Google-InspectionTool, included).

🤖 Generated with Claude Code