The AEO Technical Playbook: Everything in Execution Order

The AEO technical playbook is the ordered set of engineering and editorial steps that make a site retrievable and quotable by AI answer engines, and the order matters more than any individual step. Almost every wasted AEO budget we have seen was spent doing step five while step one was still broken, which produces exactly the same result as doing nothing.

So this is arranged as a sequence rather than a checklist you can pick from. Each step is cheap to verify before you move on, and every command and snippet below is meant to be copied and run against your own domain rather than read.

Step	What you are doing	Who owns it	Time
1	Decide crawler policy deliberately, per agent	Whoever owns robots.txt and the CDN	An hour
2	Prove the policy is what is actually being served	Engineering	An hour
3	Confirm crawlers receive content, not an empty shell	Engineering	A day to check, longer to fix
4	Make your entity unambiguous	Marketing plus engineering	A week
5	Make key passages survive extraction	Editorial	Ongoing
6	Build a measurement baseline before claiming anything	Marketing	Two weeks of runs

Step 1. Decide your crawler policy on purpose

Nine agents matter across four companies, and each company separates training from retrieval. The decision you are making is not whether to allow AI crawlers. It is whether you want to be findable without being training data, which is a supported configuration nearly everybody wants once it is explained.

Here is that configuration in full. It permits every search and retrieval agent, blocks every training collector, and leaves ordinary search engines alone.

# Retrieval and search: allowed, this is how you appear in answers
User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: PerplexityBot
Allow: /

# Model training: blocked, no effect on today's answers
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: CCBot
Disallow: /

# Everything else, including ordinary search crawlers
User-agent: *
Disallow: /wp-admin/
Allow: /

Sitemap: https://example.com/sitemap_index.xml

Three things about that file are worth understanding rather than copying blindly. Agent matching is by substring, so a rule naming ClaudeBot does not catch Claude-SearchBot, which is the entire reason the two can be controlled separately. The protocol is a published internet standard, RFC 9309, which specifies that crawlers may cache your file for up to 24 hours, so changes are not instant. And blocking Google-Extended opts you out of Gemini training while having no effect at all on Search or AI Overviews, which is the row most companies get wrong.

Step 2. Prove that policy is what you are actually serving

The file in your repository is not evidence. A content delivery network, a security product or a managed WordPress host can inject rules that never existed in your codebase, and we find this more often than we find deliberate blocks. Fetch the live version and read what the internet reads.

curl -s https://example.com/robots.txt

Then check what a specific agent receives, because a firewall can allow a URL in robots.txt and still refuse the request. Ask as the agent and read the status line and the indexing headers.

curl -sSI -A 'OAI-SearchBot/1.4' https://example.com/your-key-page/ 
  | grep -iE 'HTTP/|x-robots-tag|content-type|cf-mitigated'

A 200 is what you want. A 403 or 429 means something between you and the crawler is refusing it, and that is a firewall conversation rather than a marketing one. An x-robots-tag containing noindex is the quiet killer here, because it is invisible in the page source and instructs compliant systems to leave the page out entirely.

Now check ground truth in your access logs. This is the only measurement in the whole playbook that reports what happened rather than what should happen. Assuming a combined log format, which is nginx’s documented default, count hits by agent over whatever window you have.

AGENTS='GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-SearchBot|Claude-User|PerplexityBot|Perplexity-User|Google-Extended|Googlebot|Bingbot'

# how many hits per agent
grep -aoiE "$AGENTS" access.log | sort | uniq -c | sort -rn

# what status codes one agent is receiving
grep -ai 'OAI-SearchBot' access.log | awk '{print $9}' | sort | uniq -c | sort -rn

Zero hits for the retrieval agents means step one is not working, whatever the file says. A healthy count of 200s means you can stop worrying about access and move down the list. A wall of 403s means you have found the problem and it will take one conversation to fix.

Step 3. Make sure there is something there when they arrive

A crawler receives what the server returns before any JavaScript executes. Google renders JavaScript. Assuming every retrieval pipeline does is optimistic, and most of them publish nothing either way, so the safe assumption is the pessimistic one.

Test it without any tooling by fetching the raw HTML and measuring how much text survives.

curl -s https://example.com/your-key-page/ 
  | sed -e 's/<script.*</script>//g' -e 's/<[^>]*>/ /g' 
  | tr -s ' ' | wc -w

If that returns a number close to your actual word count, you are fine. If it returns forty, your content is being assembled in the browser and the fix is server-side rendering or prerendering. That is engineering work measured in sprints, which is why finding out now rather than after a content project matters.

The layer between steps two and three that catches people out

Your robots.txt can be perfect and your pages can render server-side, and a bot-protection product sitting in front of both can still refuse the request. This is now the most common cause of a silent block we find, because the rules were configured by a security team optimising for a different goal and nobody told marketing.

Before you write any allow rule, understand that a user agent string is trivially forged. Plenty of scrapers announce themselves as ChatGPT precisely because sites allow it. So the providers publish the address ranges their agents actually use, and matching on both the string and the range is the only reliable approach.

# the published ranges, straight from each provider
curl -s https://openai.com/searchbot.json
curl -s https://openai.com/gptbot.json
curl -s https://openai.com/chatgpt-user.json
curl -s https://www.perplexity.com/perplexitybot.json

# pull just the CIDR blocks out of one of them
curl -s https://openai.com/searchbot.json 
  | grep -o '"ipv4Prefix": "[^"]*"' 
  | cut -d'"' -f4

Feed those ranges into your firewall as an allow list for the agents you decided to permit in step one. Then re-run the log count from step two a few days later and confirm the 403s have gone. If your provider offers a managed rule set for AI crawlers, read what it actually blocks rather than trusting the label, because several of them bundle search agents together with training collectors under one switch.

One more check on the same theme. Ask for a page as an ordinary browser and then as an agent, and compare the byte counts. A large difference means something is serving different content by user agent, which is worth understanding deliberately rather than discovering later.

for UA in 'Mozilla/5.0' 'OAI-SearchBot/1.4' 'PerplexityBot/1.0'
do
  SIZE=$(curl -s -A "$UA" https://example.com/your-key-page/ | wc -c)
  echo "$SIZE bytes as $UA"
done

Three numbers within a few percent of each other is a pass. One of them coming back at a fraction of the others means you have found your problem, and it is upstream of anything an editor can fix.

Step 4. Make your entity unambiguous

This step is skipped almost universally and it is the one with the least competition. If a retrieval system cannot tell which company your name refers to, it will hedge rather than recommend, because being confidently wrong is expensive and hedging is free.

State the facts once, in machine-readable form, on a URL that never moves. The vocabulary is schema.org’s Organization type and the syntax is JSON-LD. What follows is a complete block rather than a fragment.

{
  "@context": "https://schema.org",
  "@type": "Organization",
  "@id": "https://example.com/#organization",
  "name": "Clausewise",
  "legalName": "Clausewise Software Ltd",
  "url": "https://example.com/",
  "logo": "https://example.com/logo.png",
  "description": "Contract review software for mid-market legal teams.",
  "foundingDate": "2019",
  "sameAs": [
    "https://www.wikidata.org/wiki/Q00000000",
    "https://www.linkedin.com/company/clausewise",
    "https://github.com/clausewise",
    "https://www.crunchbase.com/organization/clausewise"
  ]
}

The sameAs array is the part that matters and the part almost nobody fills in. It says this name refers to the same thing as those records, which is precisely the ambiguity problem stated in a form a machine can act on. Crawl data from the HTTP Archive Web Almanac found JSON-LD on 41 percent of pages while pages pointing sameAs at Wikidata sat at 0.17 percent, so the gap between doing structured data and doing entity resolution is enormous and it is standing wide open.

Two rules on top of the snippet. Keep @id stable forever, since it is the anchor everything else hangs off. And make sure every fact in the markup matches what a human sees on the page, because Google’s structured data policies treat markup that contradicts visible content as spam, and the penalty is worse than the omission.

Step 5. Make key passages survive extraction

Only now does writing enter the picture. Retrieval operates on fragments, so the unit of work is the paragraph rather than the page, and the test is whether a paragraph still answers something once it has been lifted away from everything around it.

Four editing rules cover most of it, and none of them require a rewrite of your site.

1. Name the subject in the sentence. Replace this, that and it with the actual noun in any paragraph you want quoted.
2. Put the number beside the claim. A threshold in the same sentence as the assertion it supports is what gets cited, because it can be checked.
3. Delete backward references. As we saw above and as discussed earlier are the two clearest signals that a passage cannot travel.
4. Answer in the first sentence, qualify in the second. Leading with context and arriving at the answer in sentence four means the answer is in a different chunk.

Test one page by hand before applying this anywhere. Read the paragraph that should win with nothing above or below it, and ask whether a stranger could act on it. Well-written prose fails this constantly, because good prose builds on itself, and that is exactly the property retrieval destroys.

Step 6. Build a baseline before you claim anything

Answers vary run to run, because the index shifts underneath you and the model samples probabilistically as it writes. One check is a sample. Two causes stacked means any single before-and-after comparison is close to worthless.

Write down ten questions your buyers ask before they know you exist. Run all ten across ChatGPT, Perplexity, Claude and Google’s AI Mode. Do it twice on different days. Record three counts with the same denominator, which are how often you were mentioned, cited and recommended. That is your baseline, and it is the thing that lets you tell a real improvement from noise three months from now.

Keep the question set frozen. If it changes between runs, the runs are not comparable, and rotating the set is the most common way visibility reporting quietly becomes fiction.

Symptom to step, for when you are debugging backwards

What you are seeing	Most likely step	The check that confirms it
Absent everywhere, including for your own brand name	Step 1 or 2	Log count returns zero for retrieval agents
Present in one engine, absent in another	Step 1, per agent	Compare robots.txt rules agent by agent
Logs show hits, but they are 403 or 429	The firewall layer	Status code breakdown for one agent
Answers describe you vaguely or wrongly	Step 4	Ask an engine what your company is
Named for your brand, absent for category questions	Step 5	Read one paragraph out of context
Cited but never recommended	None of these	Third-party corroboration, which is not technical
Results swing wildly between checks	Step 6	You are reading one sample, not a measurement

Six steps is a fortnight of work if you already have log access and an engineer with time. If you would rather find out which of them actually applies to you before committing any of that, our visibility check runs steps one through four and hands back the findings.

What to leave out

A playbook is more useful when it says what not to bother with, so here are three things sold as technical AEO that are not.

Special AI schema. There is not any. Google’s documentation states directly that no special schema.org structured data is needed to appear in its AI features, and no other engine publishes a proprietary vocabulary either. Structured data is worth maintaining for entity clarity, which is step four, and not as a ranking lever.

An llms.txt file, sold as a ranking factor. It is a community proposal rather than a standard, no provider guarantees anything about reading it, and Google says you do not need new machine-readable files. It costs almost nothing to add and it belongs in the optional column.

Rewriting your whole site into question-and-answer format. Extractability is a property of individual passages, so you get most of the benefit from editing the twenty paragraphs that matter. Converting three hundred pages into FAQ format is a large invoice for a small effect.

Frequently asked questions

Which single step matters most?

Step two, proving what you actually serve. It is the cheapest to run and the most likely to find something, because the gap between the robots.txt in your repository and the one on your domain is where the majority of real problems live.

How long before a robots.txt change takes effect?

RFC 9309 permits crawlers to cache the file for up to 24 hours, so allow at least a day before the change is even seen. Recrawling and corpus updates then take longer, and days rather than weeks is a reasonable expectation for something that was already trying to reach you.

Do I need to block training crawlers?

It is a rights decision rather than a visibility one, and it is legitimate either way. What matters technically is that training and retrieval are separately controllable, so the choice costs you nothing in answers if you make it per agent. OpenAI documents four agents and the other providers document their own.

Can I verify a crawler is genuinely who it claims?

Yes, and you should before writing firewall rules. User agent strings are trivially spoofed, so the providers publish IP ranges for their agents. Match on both the user agent and the published address range rather than the string alone.

Is any of this different for a single-page application?

Step three becomes the binding constraint rather than a checkbox. If your content only exists after JavaScript runs, server-side rendering or prerendering for the pages that matter is the prerequisite for everything downstream, and no amount of editorial work substitutes for it.

---

For why the order is what it is, the complete guide to AEO sets out the five gates these six steps are working through.

This page gets a new section each time we publish a playbook for one of these steps in depth. The newsletter is where those go out.