Bot IP lists: whose crawler it is and how to prove it

A user agent is a claim, not an identity

Every request that reaches a web server carries a line of text saying what sent it. For a browser it says Chrome or Safari; for a crawler it says Googlebot or GPTBot. That line is written by whoever makes the request, and nothing on the internet checks it. Copying it takes one keystroke.

Microsoft states the consequence plainly in its own documentation: a bot using exactly the same user agent as Bingbot does not thereby become Bingbot. So the string is useful for reading logs and useless for making decisions. Anything that grants access, skips a rate limit, or exempts a request from a rule has to be based on something the sender cannot choose — and the only thing a sender cannot choose is the address the packets actually come from.

That is the entire subject of this page: turning a name into an address you can check.

Five kinds of bot, and they are not interchangeable

Server logs mix them all together, which is why "bot traffic" as a single number tells you almost nothing. They differ in what they take and what they give back:

The AI row is the one most often collapsed by mistake. A training crawler and a retrieval crawler belong to the same company, arrive from similar addresses, and want completely different things. Blocking training costs you nothing visible. Blocking retrieval removes you from the answers that assistant gives — and that is now a real share of how people find anything.

There is a fourth thing in the logs that is not a bot at all: port scanners and vulnerability tools. They never read robots.txt, they are not crawling, and they belong in a different conversation from the one on this page.

Path one: build the list yourself, from the source

This is the slower path and the one worth understanding, because it does not depend on anyone — including this page — being honest or up to date.

Method one: the published range file

Most large operators publish the addresses their crawlers use, as a JSON file at a fixed URL. The format settled into a common shape: a creationTime and a list of ipv4Prefix and ipv6Prefix entries in CIDR notation. You fetch the file, you check whether the address in your log falls inside one of the prefixes, and you have an answer that cannot be faked.

Three rules make the difference between doing this correctly and doing it badly:

• Treat it as a feed, not as a constant. These files change. Pasting today's contents into a firewall rule produces something that is quietly wrong in a month. Refresh on a schedule and keep the previous copy if a fetch fails.
• Never substitute the hosting network for the published file. It is tempting, because the network number is easy to look up and covers everything. It is also wrong: the same cloud network hosts the crawler and every attacker who rented a machine next to it. "Verified by network number" is an answer that is confident and false.
• Use the right file for the right bot. Operators split their crawlers across several files precisely because the crawlers do different jobs. Checking a training crawler against the search crawler's file conflates two things you may want to treat differently.

One more caution that costs people hours: an address that is in the file proves the bot is genuine, but an address that is not in the file proves very little. User-triggered fetchers — the ones that run because a person asked an assistant about a specific link — legitimately come from ordinary cloud addresses outside the crawler ranges.

Method two: forward-confirmed reverse DNS

Several major search engines — Yandex, Baidu, Naver, Seznam and others — publish no range file at all, and their documentation says so explicitly: verify us by reverse DNS instead. The check has three steps and all three are required:

1. Ask what hostname the address claims: dig -x 66.249.66.1 +short
2. Check that the hostname ends in the operator's domain at a dot boundary.
3. Resolve that hostname forward — dig crawl-66-249-66-1.googlebot.com +short — and confirm the answer contains the address you started with.

Step three is what makes it "forward-confirmed", and skipping it makes the whole check worthless: anyone who controls the reverse zone for their own addresses can point it at any name they like. Only the round trip proves the operator's own DNS agrees.

Step two hides a trap that is easy to write and hard to see. Testing whether the hostname merely ends with googlebot.com also accepts evilgooglebot.com. The comparison has to be against .googlebot.com with the leading dot, or against the bare domain itself. This is a one-character bug that turns a verification into a rubber stamp.

A missing reverse record is not evidence either way. It means "not confirmed", never "confirmed fake" — plenty of legitimate infrastructure has no reverse DNS at all.

Who publishes what

The addresses below are the operators' own pages. They are printed as plain text on purpose: copy the one you need and open it yourself, so that what you end up trusting is the vendor's file rather than anyone's transcription of it — including ours.

Search engines

They send readers. Blocking one removes you from that search engine, and nothing tells you it happened.

AI crawlers

Three different jobs under one label: collecting training data, building the index an assistant cites, and fetching one page because a person asked.

Link previews

Not crawls at all. One request, made because a human pasted your link into a chat.

SEO and analytics

Commercial link indexes. They can be verified like anyone else — the question is not whether they are real, but whether they are worth the bandwidth.

Path two: take the list that is already assembled

The reference above is the honest way to do it and the slow way to do it. If you need the ranges today, here they are: one snapshot, rebuilt daily, containing every range that verification actually confirmed — from the published files above and from forward-confirmed reverse DNS for the operators who publish no files.

Snapshot built 2026-08-25T02:15:02Z (UTC). Crawler addresses change; a copy older than a week or two is already wrong at the edges.

All ranges (.txt)With bot names (.csv)Manifest (.json)

Three plain-text formats, because they get used differently. The single .txt is what you feed to a firewall or an ipset. The .csv adds the column the flat file cannot have — whose range each line is — and without it you cannot treat a search crawler differently from a training crawler. The .json is the manifest: counts per crawler and the build timestamp, for scripts that want to check freshness before trusting anything.

What being on this list means: that address really does belong to that crawler. What it does not mean: that you should let it through. Those are two different questions and only the first one has a factual answer. The second depends on what your site is for, and nobody else's list knows that.

What is deliberately missing

Some well-known bots cannot appear in any address list that is honest about its sources:

• Meta's crawlers (facebookexternalhit, Meta-ExternalAgent) publish no range file. The documented check is that the address belongs to Meta's own network, AS32934, which you can query from a routing registry — a different kind of check with different failure modes.
• Discord, WhatsApp and several other preview fetchers publish neither ranges nor reverse DNS. There is nothing to verify against. WhatsApp in particular usually draws its preview on the sender's own device, so the request arrives from an ordinary person's address and never looks like a bot at all.
• Bytespider, Cohere and most of the newer AI crawlers run from general cloud addresses with nothing published. A log line naming them is a claim and stays a claim.

Listing them anyway, with ranges guessed from observation, would be the single most damaging thing this page could do: an allow-list built on a guess is a hole that looks like diligence.

Bots people usually limit, and why

"Bad bot" flattens three unrelated problems together, and the treatment differs for each.

For the first group — the ones that behave, but cost more than they return — robots.txt is the right tool, because they read it:

User-agent: MJ12bot
Disallow: /

User-agent: BLEXBot
Disallow: /

User-agent: DotBot
Disallow: /

User-agent: Barkrowler
Disallow: /

User-agent: SEOkicks
Disallow: /

User-agent: SerpstatBot
Disallow: /

User-agent: ZoominfoBot
Disallow: /

For the second group, robots.txt is a note left for someone who does not read notes. Enforcement has to happen at the server or the edge. The rule below also repeats the names above, for the case where asking politely produced nothing:

map $http_user_agent $bad_bot {
    default 0;
    "~*bytespider"  1;
    "~*mj12bot"  1;
    "~*blexbot"  1;
    "~*dotbot"  1;
    "~*barkrowler"  1;
    "~*seokicks"  1;
    "~*serpstatbot"  1;
    "~*zoominfobot"  1;
}

server {
    if ($bad_bot) { return 403; }
}

Neither snippet helps against the last group, and it is worth being clear about why: a rule that matches on the user agent only stops the bots that tell the truth about themselves. Something pretending to be Googlebot will call itself a browser the moment the first rule bites. That is not a gap in the snippet — it is the reason the whole first half of this page exists.

Do not paste that list without reading it. facebookexternalhit appears in many published block lists, and blocking it makes every link to your site look broken in Facebook, Instagram and Messenger. Applebot is another: block it and you leave Siri, Spotlight and Safari Suggestions at the same time. The most expensive block-list entries are the ones copied without knowing what they do.

Three mistakes that cost more than the bots

Allowing a whole hosting network instead of a published range

The crawler and every attacker who rented a virtual machine beside it share that network number. Allowing it launders an impostor into a guest.

Treating "not in the list" as proof of a fake

It is not. Ranges get added; user-triggered fetchers arrive from ordinary cloud addresses by design; your copy of the file may simply be old. In range is proof of genuine. Out of range is only a reason to look closer.

Copying a list from a blog post and calling it verification

Second-hand lists are frozen at the moment someone pasted them, and they go stale without ever saying so. If the address you check against did not come from a file the operator publishes, you have not verified anything — you have agreed with a stranger.

The underlying principle is short enough to keep: a name in a log is something the sender chose, and an address is something they could not. Decide on the second one.