Agentic-ready: the four things an agent has to be able to do

In short

Four questions decide it: can an agent find the site, fetch it, read it without executing JavaScript, and recover when it hits a URL that does not exist. Most sites fail on the third for a dull reason — the content or the structured data only exists after the client bundle runs, and many fetchers never run it.

A tall door with four bolts, three drawn open and one still shut, with a small robot waiting in front of it.
The four questions
Can it be found, fetched, read, and recovered from
Most common failure
Content or schema that exists only after JavaScript runs
Second most common
Returning HTTP 200 for URLs that do not exist
This site's audit score
83 out of 100, not 100 — deliberately

"Agentic-ready" is well on its way to becoming a term that means whatever the person selling it needs it to mean. It does have a concrete definition, though, and it is not complicated: it is whether a program with no eyes, no patience and no JavaScript can accomplish anything on your site.

That resolves into four questions, in order. Failing an early one makes the later ones irrelevant.

#1. Can it find you?

The least interesting layer and the one most sites already pass. A sitemap listing every real URL with an honest last-modified date, a robots file that permits the agents you want, and internal links that make the site traversable rather than a set of islands.

One thing worth knowing: AI crawlers and AI fetchers are different agents obeying different rules. The crawler that gathers training data and the fetcher that retrieves a page because somebody asked an assistant about it are separate user agents, and permitting one does nothing for the other. If your robots file names only the famous one, you may be excluding the agent that actually matters — the one arriving because a real person asked a real question about you.

#2. Can it fetch you?

Does a plain request get a useful response — no interstitial, no consent wall in front of the content, no bot challenge, no redirect chain, correct status codes.

curl -s -o /dev/null -w "%{http_code}" https://example.com/a-path-that-does-not-exist
Should print 404. If it prints 200, this is your first job.

#3. Can it read you without running JavaScript?

This is where most sites fail, and where I failed, on the site you are reading.

Many agents do not execute JavaScript. They request the URL, take the HTML, and work with what came back. If your content is assembled client-side, they receive an empty shell. The check takes one command and it is the highest-value thing in this article:

curl -s https://example.com/ | grep -c "a distinctive phrase from your page"
If the words you expect are not in this output, an agent cannot see them.

My own version of this failure was narrower and more embarrassing, because I work in digital marketing. My pages were server-rendered and fine. My structured data was not: the identity block describing who this site is about was being injected by the client bundle rather than printed into the HTML. An audit judged the site on the one schema block it could actually see and reported the identity as missing — correctly, from where it was standing.

What "readable" means beyond merely being present:

  • Structured data as literal script tags in the served HTML, not assembled after hydration.
  • Headings that describe what follows, so the document has a real outline rather than styled text.
  • Self-contained answer blocks, because a passage that depends on the paragraph above it cannot be lifted.
  • Specifics in tables and definition lists rather than buried in prose.
  • A machine-readable summary of the site, which costs little to provide.

#4. Can it recover when it goes wrong?

The layer nobody thinks about. An agent will guess URLs, follow a stale link, or construct a path that seemed reasonable. When it lands somewhere that does not exist, what does it get?

A correct status code is the first half. The second half is giving it somewhere to go — a pointer to the sitemap, to the machine-readable summary, to whatever your index is. A 404 that only apologises is a dead end for something that cannot browse its way out.

This site's error page carries that pointer. Fitting it in was mildly annoying, because that page is a single non-scrolling screen with almost no room left, and the honest solution turned out to be showing the line only where there is space for it while keeping it in the response at every size. The markup is what a fetcher reads; the layout is what a person sees. Those are allowed to differ.

#What an actual audit found

I ran this site through a public agent-readiness audit. It scored 79, then 83 after I fixed the structured data and added the recovery pointer. It is not at 100 and I do not intend to take it there, which is the part I want to be straight about.

Two of the remaining items were declined on purpose. The largest asks for content negotiation — serving a markdown version of every page to clients that request it — and implementing that properly means generating a markdown twin of every page, plus a caching header that fragments the CDN cache and costs real visitors speed. I could not find evidence that any major AI crawler currently negotiates for markdown, and I was not willing to pay a measurable performance cost for an unmeasured benefit.

#The version to actually check

  1. A nonexistent path returns 404, not 200.
  2. Your main content appears in curl output, not only in the browser.
  3. Your structured data appears in curl output too — check separately, it fails independently.
  4. Your robots file names AI fetchers, not just AI crawlers.
  5. Your sitemap carries real last-modified dates rather than one repeated constant.
  6. Your 404 page points somewhere machine-readable.

Six commands, most of a morning. That covers the large majority of what agentic readiness actually means, and it is considerably less exotic than the phrase suggests.

Questions this answers

What makes a website agentic-ready?

Four capabilities in order: an agent can discover the site through a sitemap and permissive robots rules, fetch it without hitting walls or wrong status codes, read its content and structured data without executing JavaScript, and recover from a nonexistent URL because the 404 points at a machine-readable index.

Why does returning HTTP 200 for missing pages break AI agents?

Because it removes the agent's only signal that a URL is invalid. Single-page applications often answer every path with the app shell and a 200, so any path an agent guesses appears to exist. Without a genuine 404 it cannot distinguish a real page from a typo and may treat invented URLs as valid.

Do AI crawlers execute JavaScript?

Many do not. A substantial share of AI fetchers request the URL and work with the HTML returned, so client-assembled content and structured data injected after hydration are invisible to them. Whether content is present can be checked directly by fetching the page with curl and looking for the expected text.

See also

  • Notebook The notebook — what broke and the actual fix, plus first-person accounts of building as a marketer
  • The Story Operational experience, academic foundations and the full biography
  • All Projects The full archive of AI-native tools, systems and experiments

Let's Build
What Comes Next.

Open to meaningful collaborations, AI-native systems, product strategy, and future-focused conversations.

“Human instinct. AI amplification.
Systemic execution.”

Suman Debnath

·

Brand Marketing Leader & AI Product Builder

© 2026

This site records visit data — pages viewed, time and scroll depth, device, your IP address and the approximate location and network provider derived from it — and sends it to me privately. It also runs Google Analytics and Vercel Analytics. Full detail and how to opt out.

This site, its code and its content are © 2026 Suman Debnath. All rights reserved — none of it is open source, and copying it needs permission first. Terms of use.