What is the difference between Crawl Daddy and That One Crawler?

That One Crawler is the second version of Crawl Daddy, Suman Debnath’s SEO crawler. Crawl Daddy was a Flask server with headless Chromium on Replit that had to be republished every 30 days; That One Crawler runs on Vercel’s free tier with the crawl driven from the visitor’s browser. It keeps Crawl Daddy’s SEO score formula, so scores compare, and adds an AI-readiness score, guarded AI writing help, four focused tools and 30 guides.

How does That One Crawler score a page for AI search?

That One Crawler gives every page an AI-readiness score from 0 to 100, separate from its SEO score. Seven checks make it up: a short answer straight after the H1 (20 points), answer paragraphs under headings (20), question-style headings (15), an FAQ section or markup (15), specific facts and numbers (10), lists or tables (10), and an author and date (10).

V2 · Crawl Daddy, reprised now with fewer excuses

That One Crawler: The crawl moved into your browser.

That One Crawler is a free SEO and AI-search audit tool for blogs, built by Suman Debnath as the second version of his Crawl Daddy crawler. It crawls up to 1,000 pages, runs 47 checks on each, and scores every page twice — for search rankings and for AI-answer readiness — at $0 hosting.

Crawl Daddy, the first version, needed a server that had to be republished every 30 days to stay alive. This one turned the architecture around: the visitor’s browser drives the crawl and calls small serverless routes that each finish in seconds. Nothing to host, nothing to babysit.

47 checks per page2 scores per pageup to 1,000 pagesno signup$0 to host

🕷️ Judge a blog with V2 Meet V1, Crawl Daddy

Live · thatonecrawler.houseofnamus.com

Product, design and full-stack build · 2026 · Next.js 16 on Vercel’s free tier

I've seen things. Mostly missing H1s.

Go on, poke him

/hello-worldMissing H1
/aboutTitle too long (94 chars)
/quick-update41 words. Forty-one.
/2020-goals404, much like the goals
/gallery212 images, 0 alt tags
/best-post-evernoindex. Nobody will ever know.
/sourdough-1 + /sourdough-2Fighting over the same keyword
/old-url4 redirects deep
/2019-guideLast updated when phones had buttons
/hello-worldMissing H1
/aboutTitle too long (94 chars)
/quick-update41 words. Forty-one.
/2020-goals404, much like the goals
/gallery212 images, 0 alt tags
/best-post-evernoindex. Nobody will ever know.
/sourdough-1 + /sourdough-2Fighting over the same keyword
/old-url4 redirects deep
/2019-guideLast updated when phones had buttons

checks per page

across 9 categories, each with a guide

1,000

pages per scan

300 by default, newest posts first

400

links & images

tested per scan, beyond the pages

AI writing tasks

20 free a day, fact-checked

143

unit tests

including the site’s own copy

to host

Vercel Hobby and free API tiers

01two versions · one crawler

Same instincts. Faster legs.

Crawl Daddy came first: a Flask server on Replit that crawled with headless Chromium and had to be republished every 30 days to stay online. That One Crawler is the rebuild — the same job and the same SEO formula, with none of the babysitting, and a lot more on top. Both are live.

V1 · Crawl Daddythe original

Crawl Daddy, version one: a blue glass-style dashboard with a crawl progress bar, an SEO score of 82 and a list of issues found

Flask, Playwright and a background thread per scan. It worked — as long as somebody remembered to republish it every month.

Visit Crawl Daddy

V2 · Reprised, upgraded, on steroids

That One Crawler’s landing page: the headline “Your blog has issues. This free SEO audit finds every single one.”, the URL form and the spider mascot

Next.js on Vercel’s free tier, a second score for AI search, guarded AI fixes, four focused tools and 30 guides. It costs nothing to keep running.

Open That One Crawler

Side by sideV1 · Crawl DaddyV2 · That One Crawler

Where it ran

V1 · Replit — republished by hand every 30 days to stay online

V2 · Vercel’s free Hobby plan — every push deploys, nothing to babysit

The server

V1 · A long-lived Flask server, crawling in background threads

V2 · Small stateless API routes that each finish in seconds

Who drives the crawl

V1 · The server

V2 · The visitor’s browser — about six pages in flight

Rendering

V1 · Headless Chromium through Playwright

V2 · No headless browser. JavaScript-only pages are flagged, not rendered

Scan state

V1 · In memory, one scan at a time behind a lock

V2 · In the visitor’s browser — the last 25 scans in IndexedDB

Scoring

V1 · An SEO score out of 100

V2 · The same SEO formula, so old and new scores compare — plus an AI-readiness score

V1 · Groq and Gemini routes

V2 · Six guarded tasks, every answer checked against the page

02the key move · a browser-driven crawl

The browser runs the crawl. The server just answers.

Free serverless hosting can’t run a long background job, and a crawl of 1,000 pages is one. So the work was turned around. The visitor’s browser orchestrates discovery, crawling, link checks and PageSpeed, and calls small stateless routes that each finish in seconds. No database, no queue, no server to keep alive.

Your browser

runScan() · src/lib/client/crawl.ts

1
Discover
Follow redirects, read robots.txt, gather sitemap URLs newest first. No sitemap? Follow links from the homepage instead.
2
Crawl
Two requests at a time, three URLs each — about six pages in flight, up to the chosen limit.
3
Check links
Up to 400 links and images beyond the crawled pages: 35% internal, 40% external, 25% images.
4
Report
Scores, issues and site-wide checks are computed in the browser and saved there. Nothing goes to a database, because there isn’t one.

Stateless API routes · rate-limited per IP

/api/discover

Redirects, robots.txt, sitemap URLs newest first

20 / min

/api/analyze

Fetch and extract up to 10 pages per call

240 / min

/api/check-links

HEAD, then GET if it failed — up to 20 URLs

120 / min

/api/pagespeed

One PageSpeed Insights run

30 / min

/api/crux

Real-user speed from the Chrome UX Report

30 / min

/api/ai

One AI task, quota-checked

10 / min + daily quota

Results come back to the browser, are scored there, and are saved there: the last 25 scans in IndexedDB, on the visitor’s own device.

A scan of wordpress.org/news in progress: 15 of 25 pages read, a lime progress bar with a small spider walking it, and the verdict filling in below — thatonecrawler · scanning wordpress.org/news · 15 / 25

A 25-page scan of wordpress.org/news, captured mid-crawl. The verdict fills in while the pages arrive; this one finished in 25 seconds.

03the report · seo and ai readiness

One crawl. Two different questions.

Classic SEO asks whether a page can rank. AI search asks whether a page can be quoted. Those are different questions, so every page gets two numbers — lime for SEO, violet for AI readiness — on rings of equal size, so neither reads as the footnote.

91/ 100

SEO scoreExcellent

SEO score

Kept identical to Crawl Daddy’s formula, so old and new scores compare.

20
Has an H1
15
Title is 30–60 characters
15
Meta description is 120–160 characters
20
900+ words (10 for 300–899)
10
Has a canonical tag
5
Has an Open Graph title
5
No images missing alt text
10
Has structured data

93/ 100

AI readinessExcellent

AI-readiness score

New in V2. GEO and AEO: can an answer engine lift a clean answer from this page?

20
A 20–80 word answer straight after the H1
20
Answer paragraphs under two or more headings
15
A quarter of H2/H3s phrased as questions
15
An FAQ section or FAQPage markup
10
Three or more specific facts and numbers
10
Lists or tables
10
An author and a published or modified date

The rings show the app’s own audit of its own site: SEO 91, AI readiness 93.

47 checks, nine categories, one guide each.

Every check carries a severity, a “why it matters” and a “how to fix it”, and links to one of 30 plain-English guides. The site-wide ones — duplicates, cannibalization, orphans, click depth — can only run once the whole crawl is in, so they run in the browser at the end.

Indexing7Search appearance10Content10Links4Keywords4Structured data3AI search3Performance3Brand rules3

The finished report for wordpress.org/news: an SEO ring at 62, an AI-readiness ring at 54, the verdict “B-minus energy. Very fixable, though.” and the issue overview — the verdict · wordpress.org/news · 25 pages · 25s

The report’s Pages tab: 25 pages, each with a score, its issues as coloured chips, a status code and its title — pages · sortable, filterable, csv

The report’s Links tab: one broken external link with the page it was found on, no orphan or buried pages, and the most-linked pages — links · broken, orphans, buried, hubs

04ai help · free tiers · guardrails

AI fixes, fact-checked before you see them.

Six writing tasks — title and description, a quick-answer intro, question headings, a section rewrite, an FAQ, and a site-wide action plan. A page can contain prompt injection and a model can invent a number, so neither is trusted. Every answer is checked against the page it came from.

The AI tools tab in a page’s drawer: a title rewrite shown as a word diff, a new meta description, the Google preview, three fixes, and “Fact check passed: no invented numbers or links”, via Groq — page drawer · ✦ ai tools

Page text is fenced

Everything taken from a page is wrapped in marked blocks the model is told to treat as data, with any fake fence markers inside it neutralised. A page that tries to instruct an AI gets its own issue in the report.

Invented facts are flagged

Numbers, links and email addresses in an answer that don’t appear in the source are caught and named. When nothing is, the visitor sees “✓ Fact check passed”.

Markup is built by code

The FAQ generator writes questions and answers. The FAQPage JSON-LD around them is assembled by the app, never by the model.

Only successes count

20 free actions per visitor per day, counted by a signed HttpOnly cookie and a per-IP counter. A failed action costs nothing. After the limit, visitors can paste their own free Gemini key, which stays in their browser.

Who answers first

Title & metaGroq · short

Question headingsGroq · short

Action planGroq · short

Quick-answer introGemini · long

Section rewriteGemini · long

FAQ + JSON-LDGemini · long

Groq is fast, so short jobs go there first; Gemini has the bigger budget, so long ones do. Each falls back to the other inside a 70-second budget, and a refused key skips straight to the next provider.

05four focused tools · one engine

Four tools. Each does one job.

Not four doors into the same scan. Two check a single page in seconds; two crawl the site. They share the engine underneath, and every site crawl is saved like any scan, so the full report is one click away.

Meta tag checker

One page

Every tag search engines and social apps read, each marked good, warning or missing — beside a Google preview measured in pixels and the card a share would show.

The meta tag checker on a WordPress release post: the tag inspector beside a Google preview measured at 641px of 600px, and the social share card

GEO & AEO checker

One page

The violet ring on its own: the seven-point readiness checklist, the signals it found, and a warning if the page carries text aimed at AI assistants.

The GEO and AEO checker on a WordPress news post: a violet AI-readiness ring at 53, the verdict “Hard to quote”, and the readiness checklist

Broken link checker

Site crawl

Broken, refused-bots, errors and heavy images, each with the pages it was found on. Sites that refuse bots are listed apart, so they never inflate the broken count.

The broken link checker after 25 pages: 373 links and images tested, 1 broken, 116 sites that refused bots counted separately, and 0 heavy images

Keyword cannibalization checker

Site crawl

Pages competing for the same main keyword, grouped, with the strongest one marked to keep — best score, then most internal links, then most words.

Competing for “sourdough starter”

/sourdough-starter-guide14 links★ Keep
/my-first-sourdough3 links70
/sourdough-starter-21 links55

Illustrative pages, real shape: the strongest page is marked to keep, the others to merge, redirect or re-angle.

An honest keyword engine.

Deterministic, and it never makes anything up. Cannibalization only counts indexable pages that point their canonical at themselves.

Every keyword is a phrase that really occurs on the page. No search volumes are invented.
Plurals match: “broken links” counts toward “broken link”.
Version numbers stay whole — “WordPress 7.1.2” is one phrase, and a slug’s 7-1-2 is read as 7.1.2.
Specific beats generic: “broken link checker” wins over “broken link” when the title, H1 and body all use it.
Boilerplate can’t win: a phrase on more than 40% of a site’s pages — the blog’s name, “read more” — is nobody’s main keyword.

06engineering notes

Four problems worth writing down.

A HEAD request that lies

The constraint

Link checkers send HEAD to save bandwidth. Some servers answer every HEAD with a 404 while the page itself works fine — bsky.app is one. On a test blog that made 24 of 25 pages look like they carried broken links.

The resolution

Any failed HEAD is now confirmed with a tiny GET before anything is called broken, and 401, 403, 429, 999 and their kin are filed as “refused bots”, not broken. The same test blog went from 24 flagged pages to 1.

“A broken-link report you can’t trust is worse than no report.”

Google cuts titles in pixels, not characters

The constraint

“Keep titles under 60 characters” is a proxy. Google truncates at roughly 600 pixels, so a title of wide letters gets cut early and a title of narrow ones survives past 60.

The resolution

The preview measures each title with canvas text metrics against that limit. In the meta tag checker shot above, a 65-character WordPress title measures 641px — over by 41, and shown cut off exactly where Google would cut it.

“The preview measures what Google measures.”

A crawler anyone can point anywhere

The constraint

The form takes any URL from anyone. Without care, that is a way to make the server fetch its own internal network, follow a redirect somewhere it shouldn’t, or drain the free API quotas.

The resolution

The fetcher refuses private and internal addresses, follows at most five redirects by hand and re-checks every hop, never leaves the site being scanned, and stops at 15 seconds or 3 MB. Every API route that does real work is rate-limited per IP.

“Anyone can paste any URL. That is the threat model.”

Decorations stay out of the text

The constraint

A tool that audits headings has headings of its own. A step number or an FAQ “Q” badge written as real text becomes part of the heading — and the crawler, this one included, reads “QHow do I…?”.

The resolution

Step numbers, the FAQ badge and heading anchors are all CSS pseudo-elements. Headings read exactly as written, to search engines, to AI tools and to the app’s own self-audit.

“A crawler reads the heading you wrote, not the one you designed.”

07it practises what it preaches

It had to pass its own audit.

A tool that grades other people’s SEO doesn’t get to fail its own. The app is run against its own site as part of every check, and any new notice on its own pages is treated as a bug in the copy or in the checker.

SEO score

its own site, out of 100

AI readiness

its own site, out of 100

issues

no critical, warning or notice

pages

every URL in its sitemap

What that took

✓Every title is 30–60 characters and every description 120–160, unique across the site — enforced by tests.
✓Every guide and tool page opens with a 35–70 word direct answer, also tested.
✓A JSON-LD graph on every page, with nothing in the markup that isn’t also on the page.
✓llms.txt, llms-full.txt, a 38-URL sitemap, and a robots.txt that names the AI crawlers it welcomes.
✓Unknown guide URLs return a real 404, not a soft one.

The GEO and AEO checker run on That One Crawler’s own keyword cannibalization guide: 90 out of 100, “Ready to be quoted”, passing six of seven checks — its own GEO checker, on its own guide · 90 / 100

One of the 30 guides, on keyword cannibalization: a direct answer under the headline, a byline, and an “On this page” sidebar — one of 30 guides · direct answer, byline, on-this-page

08the voice · the spider

Brutally honest. Never mean.

An audit is a list of things you did wrong, so the app says them like a friend would. The jokes never stand in front of the information — the verdict is a joke, the issue list under it is not.

Eyes on the cursor

The spider’s pupils follow the pointer, and it blinks at slightly random intervals so a page full of spiders never blinks in sync.

It hates being poked

Poke it and it squishes and talks back. Its mood follows your score: cool, happy, smirk, meh, shocked.

Daddy mode

The Konami code — or ten pokes — swaps lime for gold and puts the shades on. A nod to Crawl Daddy.

Small rewards

Confetti for a score of 85 or more. Leave the tab and its title reads “👀 Psst… your blog still has issues”.

Honestly? Annoyingly good.B-minus energy. Very fixable, though.That’s not a URL. That’s a cry for help.Say less. Judging now.

Lines from the app. The spider at the top of this page takes ten pokes before something happens.

09constraints · honest postscripts

What it can’t do, stated plainly.

No search volumes

Keywords come from the pages themselves, and the tool says so. It never invents a volume.

Your data, your browser

No accounts, no shared report links, no scheduled emails — all three need a database. A “Time for a rescan?” nudge after 7 days and a downloadable HTML report stand in.

No headless browser

Pages that only render with JavaScript are flagged rather than rendered. A browser per scan doesn’t fit a free serverless plan.

The quota is good enough, not airtight

Clearing cookies resets the cookie count, and the per-IP counter lives in one serverless instance and resets when it restarts.

Worth admitting

The guides score 90, not 100. The SEO formula only gives full content marks at 900 words, and it was kept identical to Crawl Daddy’s on purpose so old and new scores still compare.
Titles that come back over 60 characters get one follow-up call to shorten them. The rewrite captured on this page still landed at 61 — and the card says so in its own label, “61 chars, aim for 30–60”, instead of hiding it.
A model the router depended on was retired by its provider in 2026. The router fell back to working defaults instead of breaking.
The “Reviews from the crawled” on its landing page are labelled as fiction, because they are.

Built with Claude Code. The app’s own commits carry the co-author line, and this page says so too.

10the stack

Next.js 16React 19TypeScriptTailwind CSS 4cheerioVercel HobbyGroqGeminiPageSpeed Insights APIChrome UX Report APIIndexedDBVitestBuilt with Claude Code

Paste a blog. Get judged.

Every number on this page can be checked in the live app. No signup, no email, no “book a demo” — and the original is still up, if you want to see where it started.

🕷️ Open That One Crawler V1 · Crawl Daddy

Read the notebook — where this site writes about AEO, GEO and getting quoted by AI search.

That One Crawler: The crawl moved into your browser.

Same instincts. Faster legs.

The browser runs the crawl. The server just answers.