Making a React SPA Visible to AI Crawlers Without Prerendering
GPTBot, ClaudeBot, and PerplexityBot don't execute JavaScript, so a SPA looks blank to them. How we made ours legible with build-time markdown mirrors and a request-time HTML splice, no SSR migration.
Here's an uncomfortable experiment: curl your single-page app and read what comes
back. If you're running a typical Vite/React setup, the answer is a <div id="root">,
a script tag, and nothing else. That's also exactly what GPTBot, ClaudeBot, and
PerplexityBot see, because AI crawlers don't execute JavaScript. Every docs page,
every blog post, every carefully written landing section: invisible to the systems
people increasingly use to decide what tools to try.
This site is a React SPA (it's a vibecarbon-generated app, and we build the generator,
so whatever we shipped here had to ship for every generated project too). This post is
the engineering story of making it fully legible to AI and search crawlers with no
prerendering, no SSR migration, and no framework change, and the five bugs we hit
doing it.
The constraint: keep the SPA
The stock answer is "just use SSR" or "prerender at build time." We deliberately
didn't. Migrating a working SPA to a server-rendering framework is a large, risky
change to make for an audience (crawlers) that doesn't need hydration, interactivity,
or even CSS. Crawlers need content. So we split the problem:
- Build time: generate crawler-ready artifacts from the same MDX the client renders.
- Request time: splice per-route content into the HTML shell the server already serves.
Browsers keep getting the untouched SPA. Crawlers get real HTML. Nobody gets a rewrite.
One shell, two readers. Build time fills the artifacts; request time serves each reader what it can use.
Build time: markdown is the API
A script in build:client walks the content directory (docs, blog, changelog, legal
MDX) and emits:
llms.txt: the llmstxt.org convention, a markdown index
of everything on the site, with one-line descriptions.llms-full.txt: the entire docs corpus as a single markdown file, for agents
that want one fetch.- Per-page markdown mirrors: every page also exists as
.md
(this post is readable as markdown),
served as text/markdown. LLMs parse markdown better than DOM soup, and the
llms.txt links resolve to these.
route-meta.json: a manifest mapping each route to its title, description,
canonical URL, JSON-LD, and a rendered HTML fragment of the page content.
Request time: the splice
The production server loads the built shell and the manifest once. For a known route,
it rewrites the <title> and meta/OG tags, adds a canonical link and JSON-LD, and
injects the content HTML into the root div, wrapped in a noscript element, so
browsers never paint a flash of unstyled content before hydration, while non-executing
crawlers read the text as-is. Hydration replaces the root and the browser experience is
byte-for-byte the SPA you had before.
The whole thing is ~140 lines of server code plus a generator script. No new
infrastructure, no headless Chrome at build time, no framework migration.
The five bugs worth telling you about
1. String.replace expands $ patterns in the replacement. If a page description
contains $& or $1 (think SQL docs with DO $$ blocks), JavaScript's
replace(pattern, string) expands them after your HTML escaping ran, corrupting
markup or worse. Every splice must use a replacement function, which is applied
verbatim:
out.replace(pattern, () => `${open}${escapeHtml(value)}${close}`);
2. JSON-LD can close its own script tag. A headline containing </script> ends the
JSON-LD block early and the rest parses as live HTML. Serialize with every < escaped
to the unicode form \u003c before it goes inside the script element.
3. Content can close the noscript wrapper. Same class of bug one layer up: a docs
page quoting a literal </noscript> example would end the wrapper and leak the rest
into the live DOM. Neutralize it in the injected fragment.
4. MDX prop expressions leak into crawler HTML. Our legal pages use MDX props:
the client evaluates Vibecarbon as JSX at render time. The markdown
pipeline doesn't: crawlers were reading the raw token where the company name should be.
An AI engine quoting your terms of service back with template tokens in it is a bad
look. Fix: substitute the same values at build time that the client passes at render
time.
5. Drafts leak everywhere content is enumerated. Once draft: true frontmatter
exists, five surfaces have to respect it: the route manifest, llms.txt, the
markdown mirrors, sitemap.xml, and rss.xml, plus the client's own post list.
We missed RSS on the first pass. The fix that sticks is structural: one shared
isDraft helper, and a source-level test asserting every generator uses the same
resolution rules, so the next surface can't quietly diverge.
(Bonus, because the shell is now dynamic: the injected HTML gets a memoized
content-hash ETag, so repeat crawler fetches revalidate with a 304 instead of
re-downloading every page.)
Not all crawlers are worth the same to you
One design decision that shaped the measuring side: we classify crawler hits into
ai-training (GPTBot, ClaudeBot, CCBot: corpus collection), ai-search
(OAI-SearchBot, PerplexityBot, ChatGPT-User, Claude-User: fetching a page to answer a
live question), and classic search (Googlebot, Bingbot). The distinction is the
whole point: an ai-search hit is a page being read to answer a real user right now,
the surface that can actually cite you and send traffic. Training-bot volume is mostly
noise. Vibecarbon-generated apps ship this classification with an admin page on top,
so "is AI search reading us, and which pages?" is a glance, not a log-spelunking
session.
Everything else we measure is free: a referrer segment for AI engines in analytics,
and a fixed list of prompts we manually ask the major assistants once a month. Paid
AI-visibility monitoring exists and matures fast, but it measures (it can't publish
for you), and at our stage the manual check takes twenty minutes.
Does it work?
Honest answer: we just shipped it, and we recorded the baseline first. As of this
week, we appear in none of the AI answers or search results we care about. That's the
point of this post existing: it's the before picture, published in public. We'll
report the after with the same fixed query list.
Takeaways
curl your site. What you get is what every AI crawler gets.- You don't need SSR to be crawler-legible: build-time artifacts plus a request-time
splice preserves the SPA entirely.
- Ship
llms.txt and markdown mirrors; markdown is the format AI systems actually want.
- Escape at every sink, and use replacement functions.
$ expansion after escaping is
a real injection class.
- A draft flag is only as good as its most forgotten enumeration surface. Share one
helper; guard it with a test.
The whole pipeline ships inside every Vibecarbon-generated
app (Fair Source CLI; the generated code and template are MIT). And if you build it
yourself instead: genuinely, curl it when you're done.