Twelve questions. For each one, the answer that should worry you and the answer you want. We are an agency, so read this with the obvious discount applied, and note that we have written out what we would say to each so you can hold us to the same standard.
The questions are ordered. The first four are about method, the next four about work, the last four about how it ends. If a conversation falls apart in the first four it is not worth having the other eight.
Method
1. What will you measure, and how exactly do you collect it?
You want a specific protocol. A fixed set of prompts, a named list of engines, a stated number of runs per prompt, logged-out sessions, and transcripts kept. If the answer is a proprietary score, ask how it is computed. A number nobody can reproduce is not a measurement, and the reason research fields insist on fixing the method before the results are seen is that a flexible method will always find a flattering answer.
Bad answer: we track your AI visibility score monthly. Good answer: twenty fixed prompts, four engines, two runs each, logged out, transcripts in a sheet you own.
2. How do you tell a real change from normal variance?
These systems are not deterministic. Identical prompts return different answers. Any firm reporting month over month movement without addressing this is reporting noise, and the direction of the noise will correlate suspiciously with renewal dates.
Bad answer: we use the same prompts each month. Good answer: repeated runs, and we do not call a change real unless it holds across runs and across at least two engines.
3. Who does the work, and what else are they on?
The gap between a senior practitioner doing the analysis and a junior doing it under review is roughly threefold in cost and larger than that in outcome on diagnostics. Both models are legitimate. Only one is usually disclosed. Ask for a name and ask how many other accounts that person carries.
4. What would make you tell me not to hire you?
The single most useful question on this page. A firm with a method has a clear answer, because a method produces findings that sometimes point away from the sale. A firm without one will treat the question as an objection to handle.
For the record, ours is that if your crawler access is fine, your pages are already extractable, and your problem is that nobody outside your domain writes about you, a content retainer from us is the wrong purchase and you need public relations. Our free AI visibility check reaches that conclusion for a fair share of the companies that run it.
The work
5. What would you do in the first thirty days?
You want crawler log analysis, rendering checks and a baseline. If the first thirty days are a content calendar and a keyword map, the firm has brought a search playbook to a retrieval problem.
6. Does this include implementation, or recommendations only?
Both are legitimate and the distinction is worth thousands. A recommendations-only engagement shifts the expensive part onto your engineering backlog. If that backlog is eleven weeks deep, you have bought a document and a queue position. Get it in writing either way.
7. How will you get third parties to write about us, specifically?
Corroboration is the slow half of this work and the half most often waved at. You want named tactics. Original research somebody would cite, contributed pieces, conference talks, community presence, correcting records on sites that already list you.
If the answer involves buying placements in roundups, understand what you are getting. Sponsored listings get crawled and they also get recognised as promotional, they cluster on domains whose entire business is that inventory, and the effect decays when payment stops. Disclosure rules exist for a reason, and the guidance on when a paid endorsement must be disclosed applies to your brand whether or not the agency mentions it.
8. What do you need from us, and how much of it?
Honest answer involves real effort from your side. Server log access, engineering time, subject matter experts for review, somebody to approve copy. An agency claiming it needs almost nothing from you is either underestimating the work or planning to do the kind that does not require you, which is the kind that does not move anything.
The proposal test that takes two minutes
Go through the proposal and, for each deliverable, write the constraint it addresses in the margin. Four blog posts a month addresses which constraint? If you cannot fill the margin, the plan is a production schedule rather than a strategy.
If it would help to have the constraint named and evidenced before you read any proposal, that is what our free visibility check produces. Take it to all three firms you are considering.
How it ends
9. What happens to the data if we stop working together?
Transcripts, baselines, prompt sets and tracking should be in systems you own or exportable in a usable format. If the measurement lives inside the agency’s platform, your history leaves when you do, and you cannot tell the next firm what normal looked like.
10. What is the minimum term, and what is the break clause?
Six months is a reasonable ask because this work genuinely takes a quarter to show anything. Twelve with no exit is a retention mechanism rather than a methodological necessity. Ask what happens at month four if nothing has moved, and listen for whether there is a defined answer or an awkward pause.
11. Show me an engagement that did not work and what you learned.
Everyone has several. A firm that cannot produce one either has not done enough work or is not going to be straight with you when yours goes sideways, and some of them will. The systematic review tradition treats publishing what did not work as part of the method rather than as an embarrassment, and that is the posture you want from anyone measuring anything on your behalf.
12. What security and access do you need, and how is it handled?
They will want log access, analytics, possibly CMS credentials. Ask how access is provisioned, who holds it, what happens on offboarding, and whether they have been assessed against any recognised control framework. For smaller firms a formal SOC 2 examination may be disproportionate, and in that case you want a clear written answer about named individuals, least privilege and credential rotation rather than a certificate.
What the answers tell you together
| Pattern across the twelve | What it means | What to do |
|---|---|---|
| Specific on method, vague on corroboration | Technically strong, weak on the slow half | Good for a retrieval problem. Pair with PR |
| Vague on method, strong on relationships | A communications firm using AEO language | Good for a reputation problem. Do not buy measurement from them |
| Specific on both, names a case that failed | Rare. Take it seriously | Check references and negotiate normally |
| Confident everywhere, no disconfirming answer | A sales process, not a method | Walk |
| Proposal arrived before any diagnosis | You are being sold a template | Ask for the diagnosis separately |
Run the twelve against three firms and the differences will be obvious in a way they are not from the decks. The fuller evaluation framework sits in how to choose an AEO agency, and what the answers should cost is in what AEO agencies actually charge.
One more thing worth doing with the answers. Write down, before the meetings, which three of the twelve matter most given what you already suspect about your situation. Then weight what you heard accordingly rather than counting good answers. Firms are uneven, and a firm that is excellent on the three that matter to you and mediocre on the rest is a better buy than one that is uniformly adequate.
Two questions to ask yourself instead
Before any of the twelve, work out whether you have a retrieval problem or a reputation problem, because they need different firms. A technical shop will not build you corroboration and a communications firm will not find your blocked crawler. Buying the wrong one is more expensive than paying too much for the right one. The tests are in retrieval problem or reputation problem.
And work out whether the thing you are buying is genuinely continuous. A one-off constraint deserves a project, not a retainer. If your diagnosis is crawler access plus six pages that need rewriting, that is a fortnight of internal work and no agency should be encouraging you to make it a year.
Related reading
- Agency, agency, or in-house. The comparison that sits upstream of choosing between firms at all.
- Our full testing methodology. Published so you have something concrete to compare a proposal against, including ours.
- The twenty prompt set. Run it before the first call and the conversations change completely.
Ask question four first, not last.
What would make you tell me not to hire you. It reorders the whole conversation, because a firm that has a real answer will spend the rest of the meeting diagnosing rather than pitching, and a firm that does not will spend it recovering.
When you want the diagnosis in hand before you meet anybody, book an audit call. It is free, you keep the output, and we will tell you plainly when the honest answer is that nobody needs hiring.
Shaban runs answer engine optimisation for B2B SaaS companies at Uncited Brands. He works the unglamorous end of the problem, mostly crawler access, retrieval diagnostics and measurement baselines, and publishes the tests that do not go his way alongside the ones that do.