Fetchable Is Not Cited: What You Can Actually Ship for AI Search

AEO and GEO pitches promise citations. What a site can ship is crawl and representation. This post shows where that line sits, using itself as the example.

Deep Patel

Senior Staff Software Engineer at ServiceNow

architecture

Many teams still treat “show up in ChatGPT” as a files problem. They add llms.txt, more schema, a markdown dump, and call the bundle AEO or GEO. Weeks later nobody can say what done meant, because the thing they wanted, a citation, was never something they could observe or control.

Google’s AI optimization guide, last updated July 2026, has an answer for its own surface. It names AEO (answer engine optimization) and GEO (generative engine optimization), then says that from Google Search’s perspective, optimizing for generative features is still SEO.

ChatGPT and Perplexity set their own rules, so this only settles Google. Still, the largest search surface has declined to run a separate AI ranking game. Google says Search, including AI Overviews, does not use llms.txt, extra AI markup, or markdown files. You can keep those files for other systems, and Search will neither reward nor punish you for them.

I wanted a name for the work that is left once you accept that. I call it Agent Readiness: the site is fetchable, and an agent that fetches it can represent who you are without guessing. Showing up for a query and being quoted in an answer are separate outcomes, and neither is part of it.

Five layers, two owners

It helps to draw the pipeline before arguing about acronyms.

Content, crawl, and representation are yours. Retrieval and citation belong to whoever runs the engine. Agent Readiness is layers 1 and 2, resting on content worth representing. Most AEO and GEO pitches sell layers 3 and 4 as if they were files you could add on top of 1 and 2.

What the acronyms mean

AEO started as practitioner shorthand for featured snippets and voice answers. After 2023 it stretched to cover any AI answer box. That makes it fine for conversation and too vague to define a deliverable.

GEO comes from a research paper, GEO: Generative Engine Optimization (Aggarwal et al., KDD 2024). The authors started from pages that already ranked in Google’s top five, tested rewrites such as adding quotes and statistics, and measured how much of a generated answer drew on each page. The best rewrites raised that share by up to about 40% in their experiments. The work starts after a page has been retrieved, which puts it in layer 4. It doesn’t study llms.txt, and it doesn’t promise that ChatGPT will name you.

Agent Readiness sits earlier than both. You can fail it with a Disallow, a JavaScript-only body, invented JSON-LD, or HTML and markdown that disagree. You can also pass it and still never be cited, and that is a legitimate result.

What the labs let you switch

The switch that labs document is a robots token, not a new file type. As of September 2026, the documented tokens look like this:

Each token is set independently, so allowing search while blocking training is a documented combination. Google-Extended is a token rather than a separate crawler; it covers Gemini training and some grounding products, and it does not change Search inclusion. Snippet previews are still controlled with nosnippet, data-nosnippet, and max-snippet, not a new AI file.

None of those crawler pages say “we consume your llms.txt as a ranking or citation signal.” OpenAI, Anthropic, and Gemini do publish llms.txt on their own developer docs, for agents reading those docs. Jeremy Howard’s llmstxt.org proposal (a community proposal, not an IETF standard) describes an index for inference-time agents: an H1, an optional summary, lists of links, small enough to fit in context. Version 2 also recommends publishing a companion .md copy of each page. Chrome Lighthouse marks a missing file as not applicable, but a /llms.txt that returns a 500 fails the audit, so if you ship one, it has to work.

Cloudflare’s Markdown for Agents is the other documented HTTP pattern worth copying carefully. The client requests the same URL with Accept: text/markdown, and the server answers with Content-Type: text/markdown; charset=utf-8 and Vary: Accept. Cloudflare converts HTML at the edge and has said Claude Code and OpenCode send that header. It has not said Googlebot does.

One URL is the interesting contract

This post is the example. I wrote it as a markdown file with front matter. Astro renders that file into the HTML page most people read. An agent that requests the same URL with Accept: text/markdown gets the file I wrote, byte for byte.

One trailing-slash URL carries both representations, and llms.txt at the site root lists this post as part of its index.

Why not publish the markdown as its own .md URL? A second public URL for the same page raises sitemap questions, invites guessed URLs, and a rel="alternate" link advertises a resource you may not want crawled as a page of its own. Content negotiation keeps one canonical. RFC 9110 already allows a server to ignore Accept and send HTML; a 406 is optional. Vary: Accept is how caches keep the two bodies apart. Without it, a shared cache can store whichever version the first visitor got and hand it to everyone, including the coding agent.

A post is the easy case, because the markdown is the source. There is nothing to convert, so the two versions cannot drift apart. Pages without an authored file, like my CV, get markdown generated from the same profile data as the HTML. What I would avoid is converting the rendered page back into markdown. It works until the theme toggle’s label leaks into the body.

Serving the source as-is has two side effects. The agent gets the front matter too (title, tags, draft: false), which is honest metadata, so I leave it in. The figure links also stay relative (./control-layers.webp). The HTML page publishes images under hashed build URLs, so an agent that follows those paths gets a 404. On this site the figures belong to the HTML, and the markdown promises only the text.

What a pass looks like

I run this on pateldeepp.dev:

• Crawl is allowed for everyone. llms.txt lists the five main pages, the published posts (this one included), LinkedIn, and GitHub.
• Person and WebSite JSON-LD is generated from one profile source. Home is marked up as a ProfilePage, and each post as a BlogPosting.
• Every main page and published post answers Accept: text/markdown at the same URL, through an Accept rewrite at the origin.
• Tests run against the built site and assert that there is no llms-full.txt, no public tree of .md URLs (guessing one gets a 403), and that every published post appears in llms.txt.

The JSON-LD follows the same honesty rule as the markdown. Person jobTitle and sameAs go in only if they are true and I am willing to keep them true. Google will not treat an extra alumniOf as an AI Overviews cheat code, and its guide says not to invent special generative-AI schema.

robots.txt also carries a Content-Signal: search=yes, ai-input=yes, ai-train=no line. Crawl stays allowed, and the line only states a preference, so it does not opt the site out of GPTBot. After each deploy, IndexNow pings the five main pages; posts are not in that ping set yet, and Google does not participate in IndexNow.

Other sites will need a different list. A docs site might want a separate llms.txt per section. A small personal site does not need llms-full.txt. And llmstxt.org v2’s companion .md URLs conflict with a single canonical URL if you already negotiate on Accept.

Key takeaways

1. Agent Readiness is layers 1 and 2: a bot can fetch the page, and it can rebuild the facts without guessing. Retrieval and citation belong to the engine.
2. The switch labs document is a robots token. OpenAI, Anthropic, and Google give search and training separate tokens; Perplexity’s crawler is search-only. llms.txt is neither an opt-out nor a ranking input.
3. Serve markdown at the same URL with Vary: Accept. If you already author in markdown, serve the file itself.
4. The GEO 40% is a lab number on citation share after retrieval. It is not a KPI for AI Overviews or ChatGPT search.

Where the line sits

Write fetchable and representable into the spec, and leave citation out. Then three checks are the whole deliverable:

# 1. The page a browser gets
curl -sI https://pateldeepp.dev/blog/fetchable-is-not-cited-ai-search/

# 2. The same URL, asking for markdown
curl -sI -H "Accept: text/markdown" https://pateldeepp.dev/blog/fetchable-is-not-cited-ai-search/
curl -s  -H "Accept: text/markdown" https://pateldeepp.dev/blog/fetchable-is-not-cited-ai-search/ | head -20

# 3. The index
curl -s https://pateldeepp.dev/llms.txt

The first two requests hit the same URL and should come back with different Content-Type headers, both carrying Vary: Accept. The third should list this post. If the facts in all three agree, the site is fetchable and representable. Everything past that, whether an engine retrieves the page and whether an answer names it, runs on someone else’s ranking function. I only ship the part I can test.

What I still don’t know

• Whether any search crawler has read this post’s markdown. No first-party crawler doc says it asks for text/markdown or reads a publisher’s llms.txt, and I have not set up logging that would show it.
• Whether ai-train=no is honored. Content-Signal is a stated preference. No lab has confirmed it reads the line, and the IETF AIPREF vocabulary is still a draft.
• Whether listing posts in llms.txt changes anything observable. I added them alongside this post, but I have no measurement plan yet, so I can’t claim they did anything.