Fake Googlebot: how to verify Googlebot and catch impostors
A fake Googlebot copies Google's user agent to slip past your rules. How to verify Googlebot with Google's IP lists and DNS, and what to do with fakes.
A fake Googlebot is any client that sends Google's crawler user agent without actually coming from Google. You catch it by checking the source IP, not the header: a real Googlebot request comes from an IP in Google's published crawler ranges, and its reverse DNS resolves to googlebot.com, google.com, or googleusercontent.com and back to the same IP. Everything else that calls itself Googlebot is an impostor.
Spoofing the Googlebot name is cheap, and it works surprisingly often because many sites wave anything that looks like Google straight through. This guide shows how to verify Googlebot step by step, how to find fakes in your logs, and how to handle them without hurting your search visibility.
Why bots pretend to be Googlebot
Search crawlers get special treatment almost everywhere. Teams exempt them from rate limits, challenges, and geo rules because nobody wants to block Google by accident. That exemption is exactly what a scraper or abusive bot wants to borrow.
Common reasons a bot puts Googlebot in its user agent:
- Skip rate limits and challenges. WAF and CDN rules often allow "known search engines" by user agent string.
- Read content other visitors pay for. Publishers let Google crawl paywalled articles so they can be indexed, and Google documents structured data for paywalled content for that case. A scraper that is mistaken for Googlebot can get the same access.
- Collect prices and catalogs. Ecommerce sites rarely challenge search crawlers on product and category pages.
- Hide in the logs. Requests labeled Googlebot look harmless in a quick review, so heavy crawling goes unnoticed.
Google is direct about this. Its Googlebot documentation warns that the Googlebot user agent "is often spoofed by other crawlers" and recommends verifying a problematic request before blocking it. In the bot traffic we analyze, requests that claim to be Google, Facebook, or other well-known crawlers are a recurring pattern, not an edge case.
What a real Googlebot request looks like
Googlebot comes in two main forms, Smartphone and Desktop. Google lists the user agent strings like this, where W.X.Y.Z is the current Chrome version:
Googlebot Smartphone:
Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Googlebot Desktop:
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Chrome/W.X.Y.Z Safari/537.36
A fake can copy these strings exactly. That is why the user agent only tells you what the client claims to be. Two things it cannot easily fake are where the request comes from (the IP) and who controls the DNS for that IP.
How to verify Googlebot: the two official methods
Google documents two ways to verify its crawlers. Use the first for automated checks at scale and the second for one-off investigations or when an IP is missing from the lists.
Method 1: match the IP against Google's published ranges
Google publishes its crawler IP ranges as JSON files in CIDR format:
| File | Covers |
|---|---|
common-crawlers.json | Common crawlers such as Googlebot |
special-crawlers.json | Special-case crawlers such as AdsBot |
user-triggered-fetchers.json and user-triggered-fetchers-google.json | Fetchers that run because a user asked for them |
user-triggered-agents.json | User-triggered agents |
A practical setup:
- Download the files on a schedule (daily is a reasonable default) and keep the ranges in memory or in a fast lookup table.
- For every request whose user agent claims to be a Google crawler, check whether the source IP falls inside one of the ranges.
- Treat a match as verified. Treat a miss as unverified and either run the DNS check below or apply your normal bot rules.
This check is fast and needs no network call on the request path, which matters on busy sites.
Method 2: reverse DNS, then forward DNS
For a single IP from your logs, Google's manual check has four steps:
- Run a reverse DNS lookup on the IP.
- Confirm the hostname ends in
googlebot.com,google.com, orgoogleusercontent.com. - Run a forward DNS lookup on that hostname.
- Confirm it returns the original IP.
With the host command, a real Googlebot looks like this (example from Google's documentation):
$ host 66.249.66.1
1.66.249.66.in-addr.arpa domain name pointer crawl-66-249-66-1.googlebot.com.
$ host crawl-66-249-66-1.googlebot.com
crawl-66-249-66-1.googlebot.com has address 66.249.66.1
Both steps matter. Whoever controls an IP block also controls its reverse DNS, so they can make it return a hostname that ends in googlebot.com. The forward lookup closes that gap, because only Google controls what googlebot.com names resolve to.
What about Web Bot Auth?
Google has started to experiment with Web Bot Auth, an IETF draft in which bots sign their requests cryptographically. For now Google signs only a subset of requests from Google-Agent, and it recommends continuing to rely on IP addresses, reverse DNS, and user agent strings. Signed requests are worth watching, but they do not replace the checks above yet.
How to find fake Googlebots in your logs
You do not need a special tool to get a first picture. A quick audit:
- Filter requests that claim to be Googlebot. For example,
grep -i "googlebot" access.log. - Group them by source IP and count requests. Real Googlebot traffic comes from a fairly stable set of Google networks; fakes often come from hosting providers, VPNs, or residential proxies.
- Verify the top IPs against the JSON ranges or with the DNS check.
- Look at what the unverified ones requested. Paths like search, pricing, APIs, or the full archive at high speed are typical for scrapers. A real crawler usually follows links and sitemaps.
- Check the timing. Google says Googlebot should generally not access a site more than about once every few seconds on average. Sustained bursts from one IP under the Googlebot name are a strong hint.
Other crawlers can be checked the same way. Bing, for example, publishes bingbot.json and documents a reverse DNS check that ends in search.msn.com on its webmaster blog.
What to do with a fake Googlebot
Once a request fails verification, it is simply an unidentified client that lied about its identity. That lie is a useful signal in itself.
- Do not give it crawler privileges. No rate-limit exemption, no paywall bypass, no skipped challenges.
- Block or challenge on sensitive routes. Login, search, APIs, checkout, and premium content are where fake crawlers do the most damage.
- Rate-limit elsewhere. If you are unsure, slowing a client down is less risky than blocking it.
- Log it as spoofed, not just "bot". A separate label shows you how often it happens and which paths it targets.
- Stop exempting by user agent. If your WAF or CDN allows "Googlebot" based on the header alone, change the rule to require a verified IP.
If the real Googlebot is putting too much load on your servers, the answer is different. Google recommends returning 500, 503, or 429 for a short time (a couple of hours to one or two days) to reduce the crawl rate, and warns that URLs returning these codes for several days may drop out of the index. Blocking real Google crawlers outright will hurt your search visibility.
Common mistakes when verifying Googlebot
- Trusting the user agent alone. It is the one thing every fake gets right.
- Doing only the reverse lookup. Reverse DNS records are set by whoever controls the IP; the forward lookup is what proves the hostname belongs to Google.
- Running DNS on every request. On high-traffic routes, use the IP lists first and do DNS checks asynchronously for IPs that are not on the list. We describe this pattern in more detail in our guide to recognizing good bots.
- Hard-coding old IP lists. Google's ranges change. Refresh them automatically.
- Forgetting other Google fetchers. AdsBot, user-triggered fetchers, and agents use different lists and hostnames. A request can be from Google without being Googlebot.
- Treating static IP blocklists as the fix. Fakes rotate through many networks, which is why IP lists and firewalls alone are often not enough.
Where Atmosvere fits
Verifying Googlebot is a solvable problem on its own. It gets harder when you need the same discipline for dozens of crawlers, AI agents, and partner integrations, each with its own lists and naming, and when you want decisions per route instead of a single allow rule.
Bot Protection in Atmosvere Bot Shield checks claimed crawler identities against network ownership on every request and labels spoofed crawlers separately from verified ones, so a fake Googlebot is treated as an unknown client rather than as Google. Verified crawlers stay visible and keep their access. For publishers, Crawler Management adds the policy layer on top: which verified search and AI crawlers to allow, which to limit, and which to block.
Bottom line
The Googlebot user agent proves nothing. Verify by IP, using Google's published ranges first and reverse plus forward DNS as the fallback, refresh the lists automatically, and never grant crawler privileges to a request that fails the check. Real Googlebot keeps crawling, and the impostors lose the free pass.
Want to know how many of the "Googlebots" on your site are real? Request a free bot analysis.
Common questions
- Can I trust a request just because the user agent says Googlebot?
- No. The user agent is a header the client chooses, and Google itself notes that the Googlebot user agent is often spoofed. Only the source IP proves origin: check it against Google's published IP ranges or with a reverse DNS lookup followed by a forward lookup.
- Will blocking fake Googlebots hurt my SEO?
- Not if you verify before you block. A request that claims to be Googlebot but comes from an IP outside Google's ranges and fails the DNS check is not Google, so blocking it does not affect crawling or indexing. The risk is blocking by user agent or by guesswork, which can catch the real crawler.
- Which domains does a real Google crawler resolve to?
- According to Google's documentation, reverse DNS for Google's crawlers and fetchers ends in googlebot.com, google.com, or googleusercontent.com. The forward lookup of that hostname must return the same IP you saw in your logs.
- What is Web Bot Auth and does it replace IP checks?
- Web Bot Auth is an IETF draft for cryptographically signed bot requests. Google describes its support as experimental and so far signs only a subset of Google-Agent requests, so it recommends continuing to rely on IP addresses, reverse DNS, and user agent strings.
