SEO & AI Search
Serve real HTML to AI crawlers and search engines, and publish llms.txt artifacts for AI agents.
Overview
AI crawlers (GPTBot for ChatGPT, ClaudeBot, PerplexityBot) and Bingbot fetch raw HTML and never execute JavaScript. A plain single-page app returns an empty <div id="root">, so to those crawlers every page on your site looks blank.
Your app solves this without prerendering React. In production, the server injects per-route content and metadata into the SPA shell at request time:
<title>, <meta name="description">, and the matching Open Graph / Twitter tags<link rel="canonical">- JSON-LD structured data (
TechArticle for docs pages, BlogPosting for blog posts)
- The page's actual content as HTML, spliced into
#root
Browsers get the same document, then React hydrates and replaces #root, so the SPA behaves exactly as before. Only the first raw HTML response differs.
Generated Artifacts
npm run build runs scripts/generate-seo.ts, which reads your MDX in content/ and emits:
| Path |
Contents |
/llms.txt |
Index of your docs, blog, and changelog for AI agents (llmstxt.org convention) |
/llms-full.txt |
Every docs and blog page concatenated into one markdown file |
/docs/<slug>.md |
Per-page markdown mirror (also /blog/<slug>.md, /changelog/<slug>.md) |
The markdown mirrors are what the links in llms.txt point at, so an agent that follows the index lands on clean markdown rather than HTML.
The same script writes the route metadata the server injects, so all four outputs stay in sync with your content automatically. Adding an MDX file is enough. Nothing needs registering.
Customizing
Per-page title and description
The title and description in each MDX file's frontmatter are the single source for both the client-side <SEO> component and the server-injected metadata:
---
title: "Billing"
description: "Set up plans, checkout, and the customer portal."
order: 11
---
Write descriptions as one plain sentence: they appear in search results, in llms.txt, and in the injected <meta> tags.
Homepage summary
The homepage has no MDX file, so its crawler-visible summary is a block of HTML in scripts/generate-seo.ts. Edit it when you change your hero copy so the two don't drift apart.
robots.txt
src/client/public/robots.txt allows all crawlers, including the AI ones, and disallows only the authenticated routes (/dashboard, /settings, /admin, /api/). Blocking an AI crawler here removes you from that assistant's answers. Leave the wildcard rule alone unless that's what you want.
The build rewrites the Sitemap: line to your real site URL and generates sitemap.xml alongside it.
Getting Indexed
On-site markup only matters once crawlers find you.
- Submit your sitemap to Bing Webmaster Tools. ChatGPT retrieves pages through Bing's index, so Bing coverage is the practical prerequisite for showing up there.
- Submit it to Google Search Console as well, for Google and Gemini.
- Third-party mentions outweigh on-site tweaks. Assistants cite what other sites say about you. Documentation on your own domain rarely moves the needle by itself.
AI Visibility dashboard
Once the crawlers arrive, Admin → AI Visibility shows what they did. The server classifies every document request by user agent against a built-in crawler registry (GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Bingbot, Googlebot, Amazonbot, CCBot, and friends) and records the hit. Asset and API requests are skipped, so the numbers reflect pages, markdown mirrors, and llms.txt rather than JavaScript bundles.
The page answers the two questions worth acting on:
- Is anything crawling you at all? Totals per crawler and per category (AI search, AI training, search engine), plus a daily trend chart.
- What are they actually fetching? A most-crawled-pages table. If your pricing page never appears but your changelog does, that's a content gap, not a markup bug.
Only the crawler name, the requested path, a truncated user-agent string, and a timestamp are stored (no IP addresses), so this adds no personal-data surface. Raw hits are kept for 90 days and rolled up nightly into a daily table that's retained indefinitely; both jobs run on pg_cron and can be triggered by hand from Admin → Jobs (rollup-crawler-hits and cleanup-crawler-hits).
An empty dashboard usually means the site isn't reachable yet rather than that something is broken: crawlers can only visit a public domain, and robots.txt has to let them in.