Good bots explained: how to recognize them, robots.txt, and why you still need visibility

Good bots are automated clients you often want on your site: search crawlers, uptime monitors, payment or partner integrations, and some AI assistants that fetch pages with clear provenance. The challenge is telling them apart from scrapers and abuse — and governing both with policy, not vibes.

If you run a European media site, marketplace, or SaaS, good-bot traffic affects SEO, availability, and infrastructure cost. Ignoring it is as risky as blocking it blindly.

What counts as a “good” bot?

“Good” is a business judgment, not a moral label. Typical categories:

For deeper product context, see crawler management and scraper protection.

How to recognize good bots (beyond the User-Agent)

1. Declared identity (User-Agent)

Cooperative crawlers usually identify themselves. Examples you will see in logs (strings evolve — treat these as patterns, not a forever allowlist):

# Search
Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)
Mozilla/5.0 (compatible; DuckDuckBot-Https/1.1; https://duckduckgo.com/duckduckbot)
Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)

# Previews / unfurls
facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)
Twitterbot/1.0
LinkedInBot/1.0 (compatible; Mozilla/5.0; Apache-HttpClient +http://www.linkedin.com)

# Monitoring (vendor-specific; examples)
Mozilla/5.0 (compatible; Pingdom.com_bot_version_check)
Mozilla/5.0+(compatible; UptimeRobot/2.0; http://www.uptimerobot.com/)

# AI / assistant crawlers (policies differ by vendor — verify current docs)
Mozilla/5.0 (compatible; GPTBot/1.0; +https://openai.com/gptbot)
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)
Mozilla/5.0 (compatible; Bytespider; spider-feedback@bytedance.com)

Important: Anyone can set User-Agent: Googlebot. Recognition requires verification, not string matching alone.

2. Network verification (IP lists first, DNS as fallback)

For major search crawlers, do not reverse-DNS every request on the hot path. Prefer this order:

1. Primary: known crawler IP lists / ownership ranges
   Match the connecting IP against maintained ranges (vendor-published prefixes, platform threat intel, or your own allowlist of verified crawler nets). A hit is fast, cache-friendly, and enough to treat the client as a candidate good bot — still combine with User-Agent expectations and route policy.
2. Fallback: DNS only when the IP is missing
   If the IP is not on the list, then run the classic verification chain:
   • Reverse DNS on the connecting IP
   • Confirm the hostname belongs to the crawler’s domain (e.g. *.googlebot.com)
   • Forward-confirm DNS back to the same IP
     On success, add or refresh that IP (or its prefix) in your list so the next hit is an IP lookup again — not another DNS round-trip.
3. On high-traffic routes: DNS must be async
   Login, APIs, checkout, and other hot paths cannot wait on DNS. If the IP misses the list, make a provisional decision from other signals (challenge, soft allow with monitoring, or default deny on sensitive routes), and complete reverse/forward DNS asynchronously. Promote verified IPs into the primary list offline; never block the request on a synchronous DNS timeout under load.

If the IP list misses and the DNS chain fails (or never confirms), treat the client as spoofed — even if the User-Agent looks perfect.

3. Behavior on your routes

Good bots tend to:

• Fetch publicly linked pages and assets at a steady pace
• Honor (or at least attempt) crawl-delay / robots rules when they claim to
• Avoid credential stuffing patterns on /login and /signup
• Show consistent ASN / IP ownership over time

Abusive automation often concentrates on inventory, pricing APIs, forms, or account endpoints — see our primer on what bot traffic is.

robots.txt: still worth doing (even when some ignore it)

robots.txt is a voluntary convention at your site root (https://example.com/robots.txt). Cooperative crawlers read it; many scrapers and custom scripts do not.

Use it anyway:

• It documents intent for legitimate crawlers and AI bots that honor it
• It reduces accidental overcrawl of staging paths, faceted URLs, and thin parameters
• It is not access control — sensitive data needs auth, not a Disallow line

Minimal example

# https://www.example.com/robots.txt
User-agent: *
Disallow: /admin/
Disallow: /api/internal/
Disallow: /cart
Disallow: /*?*sort=
Allow: /

# Example: be explicit for a named crawler (check that vendor’s current product name)
User-agent: GPTBot
Disallow: /private/
Allow: /blog/

Sitemap: https://www.example.com/sitemap.xml

Tips that age well:

• Keep one canonical robots.txt on the production host
• Point to your XML sitemap
• Prefer specific rules over a blanket Disallow: / unless you truly want to de-index
• Revisit AI crawler tokens as vendors rename agents — policies change faster than search bots

When a client ignores robots.txt, that is a classification signal for bot protection: cooperative vs non-cooperative automation.

Why a bot management tool should analyze good bots too

Allowlisting “Googlebot” in a WAF and forgetting the rest leaves blind spots. A proper Bot Shield-style control plane helps because good bots still:

1. Cost money — Crawl volume drives CDN, origin CPU, and cache churn
2. Shape SEO and previews — Mis-challenging a real crawler hurts indexing and unfurls
3. Overlap with scrapers — Same paths, similar concurrency; you need labels, not just blocks
4. Change over time — New AI agents appear; verification rules and allowlists need updates
5. Inform policy — You may allow Googlebot on /, throttle aggressive AI crawl on /pricing, and block spoofed UAs on /login

Without analytics that separate good, bad, and unknown automation, teams either over-block (false positives) or under-protect (silent scrape and abuse). Visibility turns robots.txt and User-Agent folklore into measurable policy: allow, challenge, or block per route.

A practical setup checklist

1. Publish a clear robots.txt + sitemap; document AI crawler intent
2. Verify major crawlers with IP lists first; DNS only on missing entries — never by User-Agent alone
3. On high-traffic routes, keep DNS async so verification does not add latency
4. Allowlist known monitors and partners on health-check paths only
5. Keep login, signup, and payment routes on stricter sensitivity
6. Review bot reports weekly: volume by class, robots.txt violators, spoofed UAs
7. Prefer adaptive friction over permanent puzzles for ambiguous traffic (CAPTCHA alternative)

Bottom line

Good bots are real, useful, and sometimes expensive. Recognize them with identity + IP-list verification (DNS only on misses, async on hot paths) + behavior, publish robots.txt for cooperative clients (knowing others will ignore it), and use bot management to analyze and govern legitimate automation — not only to stop the bad kind.

Want help classifying crawlers on your stack? Book a demo or request a free bot analysis.