Two crawlers wore OpenAI's name and asked for my .env file

This site has recorded 7,214 crawler visits. Of those, 530 arrived with a user agent that says it is automation but names no product, and 17 arrived with no user agent at all. The other 6,667 all named themselves, and naming yourself costs nothing.

On 1 September two of them named themselves OpenAI.

#Two requests wore OpenAI's user agent and came from the wrong cloud

They carried the exact ChatGPT-User string OpenAI publishes, and they asked for /.env.sample and /.git/HEAD. Both returned 404 — nothing was exposed, and .env* has never been committed to this repository. But the alerts reported both of them as "OpenAI · ChatGPT live fetch", which is what the user agent said and what nothing else had checked.

They came from 104.23.175.224 and 162.159.98.239. Both are Cloudflare. Every one of the 204 prefixes OpenAI publishes for that agent is Azure. The two facts do not sit together, and the only reason I know that is that OpenAI publishes the list at all.

There is a real objection to that, and it is worth putting before the argument rather than after it. ChatGPT-User is a user-directed fetcher — it fires when somebody asks the assistant to open a specific URL, rather than on a crawl schedule. Fetches of that kind can legitimately arrive from general cloud infrastructure outside the published ranges, which means an address outside the list is not on its own proof of anything.

What settles it here is the second half. Nobody asked an assistant to open /.git/HEAD on a personal portfolio. The pair of requests only makes sense as a scan, and the user agent was chosen because it gets waved past filters that stop curl. Take the paths away and I would have had a suspicious address and no conclusion.

#A forged crawler corrupts the one number the dashboard exists to produce

The obvious reading of this is a security one, and it is the reading everybody writes. Somebody probed for secrets, the probe failed, move on.

The reading that cost me more is different, and it arrived late: I did not build the crawler feed to catch intruders at all. I built it to answer one question. Are answer engines reading this site? That number is the entire point of the answer-engine work — it is how you tell being crawled from being ignored, months before anything shows up in an answer.

A forger inflates exactly that number. Two visits reported as ChatGPT live fetches are two visits of evidence that OpenAI is paying attention, and they were nothing of the kind. Whoever sent them was not trying to corrupt my measurement — they wanted my environment file. The corruption was free.

Which means the metric that flatters you is the one to check first, and a crawler log with no verification in it is a list of things clients said about themselves.

#Only five of the crawlers hitting this site can be checked at all

Verification works by asking the vendor. OpenAI, Google, Microsoft, Perplexity and Anthropic each publish a machine-readable file of the IP ranges their crawlers use. You take the client address, and you ask whether it falls inside one of the prefixes in that vendor's file.

Those five are the ones my verifier reads, which is not the same as the ones that exist — Apple, DuckDuckGo and Common Crawl publish lists too, and mine does not fetch them. So that count describes my code, not the field.

Five vendors against 7,214 visits is the honest shape of this: verification is not a filter you apply to your traffic but a check available on a small, named minority of it, and the correct answer for everything else is that you do not know.

An IP check is also the weakest of the three methods that exist, and I should say so. Forward-confirmed reverse DNS resolves the address to a hostname, checks the hostname belongs to the vendor's domain, then resolves that hostname forward again to confirm it lands back on the same address — the forward step is what defeats a forged PTR record. Web Bot Auth goes further and has the crawler sign each request with an Ed25519 key, which needs no IP list at all. Mine does neither. It reads a JSON file and compares prefixes, which catches an opportunistic scanner and would not trouble anybody serious.

#A wrong "forged" costs more than a wrong "verified"

The check has three possible answers, not two: verified, forged, and unverified. Getting the third one right is most of the work.

"Forged" is an accusation. It should only ever be made from a list that actually loaded and actually parsed. A network failure, an empty prefix array, a vendor that publishes nothing, a missing or unparseable client IP — every one of those has to come back "unverified".

The asymmetry is deliberate and it is worth stating as a rule. A wrong "verified" costs one missed scanner. A wrong "forged" teaches you to distrust the alerts, and once you distrust them you stop reading them, which costs the whole feature. So the failure modes are not symmetrical and the code should not treat them as though they were.

#Anthropic had published its list three weeks before I looked

I wrote the verifier on 1 September and put a comment in it saying Anthropic published no such list, so Claude-User, Claude-SearchBot and ClaudeBot would always come back unverified. I had tried claudebot.json, claude-user.json and ips.json on anthropic.com — all 404 — and a path on docs.claude.com that answers 200 with HTML.

The list is at `claude.com/crawling/bots.json`. Its creationTime reads 18 August 2026 — two weeks before I sat down and concluded it did not exist, and every Claude visit in the three weeks after that was filed unverifiable on the strength of one comment I never went back to.

Four addresses I made up returned 404, and I turned that into a claim about what a company publishes. A 404 on a path you invented is not evidence that a vendor publishes nothing. It is evidence that you guessed four times and missed.

#Anthropic publishes no IPv6 ranges, so a real Claude fetch can look forged

Wiring the list in is where the second trap is, and it is the one that would have done real damage. Anthropic's file carries 26 prefixes and every one of them is IPv4. There is not a single ipv6Prefix entry in it.

A correct matcher refuses to compare a v4 address against a v6 prefix — otherwise a v4 address falls inside a wide v6 range by accident, because 32-bit values sit at the bottom of the 128-bit space. So a genuine Claude fetch arriving over IPv6 matches nothing in Anthropic's file, falls past the end of the loop, and reaches whatever the function returns last.

In mine, what it returned last was forged. Hosting hands over IPv6 for a growing share of clients, so the naive version of this feature would have started accusing Anthropic of impersonating itself — silently, and with a confident label on the alert.

The fix is a guard before the matching loop: if none of the vendor's loaded lists holds a single prefix of the client address's family, return unverified with the reason, and never reach the accusation. It is four lines, it is not specific to Anthropic, and any single-family vendor list has the same hole waiting in it.

Both traps have the same shape, which is why they are worth ending on. Neither one produces an error. A soft-404 parses to zero prefixes and every crawler becomes a liar; a missing address family matches nothing and the honest one becomes a liar. In each case the dashboard keeps working, the alerts keep arriving, and the labels on them are wrong.