We use cookies and similar technologies to analyze how visitors use this website so we can improve our product and your experience. See our cookie policy.

Trusted Accounts is now Atmosvere

BlogBot protection

Anti-scraping: how to prevent website scraping, step by step

By Ludwig Thoma

Anti-scraping works in layers: limit what you expose, rate-limit, verify crawlers, and classify behavior. A practical guide to prevent website scraping.

Anti-scraping means making automated copying of your content and data hard, expensive, and visible, without getting in the way of real visitors or the crawlers you want. No single control does that. You prevent website scraping with layers: expose less data, rate-limit, verify crawlers that claim a known identity, classify client behavior, and challenge or block what remains on the routes that matter.

This guide covers how to tell whether you are being scraped, which anti-scraping measures stop which kind of scraper, and the mistakes that make protection either useless or hostile to users.

What counts as scraping

OWASP lists scraping as an automated threat (OAT-011) and defines it as collecting application content and other data for use elsewhere. That definition is broad on purpose. Search engines also collect content, so "scraping" is less about the technique and more about whether you agreed to it.

Typical scrapers you will meet:

WhoWhat they wantWho it hurts most
Competitors and price intelligence toolsPrices, stock levels, product catalogsEcommerce
Content republishers and spam sitesFull articles, often republished within minutesNews and media
AI crawlers and data collectorsText at scale for training or answersNews and media, any content site
Data resellersListings, profiles, reviewsMarketplaces, platforms
Account and API abusersData behind logins or in internal APIsSaaS, platforms

Some AI crawlers identify themselves and respect robots.txt; others do not. Declared crawlers are a policy question (allow, limit, or block), while undeclared ones are a detection problem.

Why scraping is worth stopping

  • Lost exclusivity. A paywalled or exclusive article republished elsewhere loses value, and your copy competes with duplicates.
  • Lost margin. Competitors who scrape prices around the clock can undercut you automatically.
  • Infrastructure cost and outages. Scrapers often request uncached pages, search results, and filters. Heavy scraping waves slow sites down and can take them offline.
  • Distorted data. Scrapers that run a real browser can show up in analytics as visitors, which skews traffic numbers and conversion rates.
  • Weaker negotiating position. Publishers who cannot see who crawls their content have no basis for licensing conversations with AI companies.

How to tell you are being scraped

Look for these signals in server logs, CDN logs, and analytics:

  1. Sequential or exhaustive access. Product IDs, article archives, or paginated lists requested in order, often far deeper than humans ever browse.
  2. High request rates on expensive routes. Search, filters, price endpoints, and APIs with much more traffic than the pages that link to them.
  3. Traffic without the usual assets. Many HTML requests but few CSS, image, or font requests from the same client.
  4. Clusters from hosting networks. Lots of traffic from cloud and data center ASNs, or from residential proxy ranges, with similar timing.
  5. Suspicious crawler claims. Requests that say they are Googlebot or another known crawler but come from the wrong networks. See how to verify Googlebot.
  6. Your content elsewhere. Exact copies of new articles or full catalog data appearing on other sites shortly after you publish.

Anti-scraping measures that work, layer by layer

Each layer stops a different kind of scraper. Combine them; do not pick one.

1. Publish your rules, but do not rely on them

Use robots.txt and your terms of use to state what automated access you allow. Cooperative crawlers follow robots.txt, and clear terms help in legal and licensing discussions. But the robots.txt standard, RFC 9309, says plainly that its rules "are not a form of access authorization." Treat it as a statement of intent, not a lock.

2. Expose less data

The cheapest scraping defense is not handing out data in bulk:

  • Do not send full article text, full price lists, or internal fields in JSON that the page does not display.
  • Cap page sizes in APIs and paginated lists, and avoid endpoints that return everything at once.
  • Keep internal and partner APIs authenticated, with per-key quotas.
  • Serve premium content only after the server has checked access, not hidden in the page with CSS or JavaScript.

3. Rate-limit per client, not just per IP

Set limits on expensive routes (search, filters, APIs, archives) and return 429 Too Many Requests, the status code defined in RFC 6585 for this purpose. Limit per IP, per session, and per account where you have them. Per-IP limits alone are easy to evade with proxy pools, so watch for many IPs that share the same behavior.

4. Verify crawlers that claim a known identity

Scrapers often pretend to be Googlebot, Bingbot, or a social preview bot because those get special treatment. Check claimed identities against the IP ranges the operators publish, and fall back to reverse and forward DNS where they document it. A client that fails verification should lose every privilege the real crawler has. Our guide to recognizing good bots covers how to do this without slowing down your site.

5. Classify clients by behavior and signals

Modern scrapers run headless browsers, rotate residential IPs, and copy real browser headers. Detecting them takes a combination of signals rather than one rule:

  • Consistency between claimed browser, TLS and HTTP characteristics, and JavaScript environment
  • Navigation patterns: entry points, depth, speed, and order of requests
  • Network reputation and ownership of the source IP
  • Session behavior over time, not just single requests

This is the core of bot management, and it is where most homegrown rule sets stop scaling.

6. Challenge only what is ambiguous

Put a challenge in front of traffic you cannot classify with confidence, not in front of everyone. Blanket CAPTCHAs frustrate readers and buyers, and solver services get past many of them anyway; we explain the limits in why reCAPTCHA is not enough for bot management.

7. Monitor and adapt

Scrapers change tactics when they get blocked: new IP pools, slower rates, different entry points. Review what is being blocked and what gets through every week, and adjust thresholds and rules. Anti-scraping is not a one-time setup.

Which layer stops which scraper

Scraper typeRules (robots.txt)Expose lessRate limitsCrawler verificationBehavior classification
Declared, cooperative crawlerYesPartlyYesNot neededNot needed
Simple script from one serverNoPartlyYesNoYes
Fake Googlebot or other spoofed crawlerNoPartlyPartlyYesYes
Headless browser on rotating residential IPsNoPartlyRarelyNoYes
Logged-in account abuseNoYesPer accountNoYes

Common anti-scraping mistakes

  • Blocking all bots. You lose search visibility, link previews, and monitoring. Decide which automation you want, verify it, and act on the rest.
  • Relying on IP blocklists. Lists go stale within days, and scrapers rotate through networks faster than you can update them.
  • Obfuscating HTML. Scrambled class names or text rendered as images hurt accessibility and SEO more than they hurt scrapers.
  • Protecting only the homepage. Scrapers go where the data is: search, product pages, archives, and APIs.
  • Setting it and forgetting it. Rules that worked last quarter may let today's scrapers through.

Publishers and ecommerce: what to prioritize

News and media: Start with visibility. Know which crawlers read your articles, which ones are verified, and how much they take compared with the readers they send back. Then decide per crawler: allow it, limit it, or block it. Protect the article archive and any full-text feeds and APIs, and verify every client that claims to be a search crawler.

Ecommerce: Start with price, stock, and search endpoints. Rate-limit filters and search, keep product APIs authenticated, and watch for sequential crawling of product IDs. Scraping waves that slow your shop down cost conversions even when nothing is stolen.

Where Atmosvere fits

The measures above can be built in-house, and the first three layers should be part of any setup. Layers four to seven are where most teams need dedicated tooling, because they depend on signals across many requests and on keeping detection current.

Atmosvere Scraper Protection, part of Bot Shield, classifies traffic on every request, separates verified crawlers and partners from spoofed and unknown automation, and lets you allow, challenge, or block per path. For publishers, Crawler Management turns AI crawler access into a business decision: allow, require payment, or block. Atmosvere is an Austrian company and runs on EU-owned infrastructure, so the traffic data used for these decisions stays under European control.

Bottom line

You cannot make a public page impossible to copy, but you can make bulk scraping slow, costly, and visible. Expose less, rate-limit expensive routes, verify crawlers that claim a known identity, classify behavior, and challenge only what stays ambiguous. Review the results regularly, because scrapers adapt.

Want to see who is scraping your site today? Request a free bot analysis.

Common questions

Can you completely prevent website scraping?
No. Anything a browser can display, a determined scraper can eventually copy. The realistic goal is to make large-scale, automated copying slow, expensive, and visible, while real visitors and the crawlers you want keep full access.
Does robots.txt stop scrapers?
No. robots.txt states rules that crawlers are asked to follow, and the standard (RFC 9309) says explicitly that these rules are not a form of access authorization. Cooperative crawlers respect it; scrapers that want your data usually ignore it.
Is rate limiting enough to stop scraping?
Rate limiting stops simple scripts that hammer your site from one IP. Scrapers that spread requests across thousands of residential or cloud IPs stay under per-IP limits, so you also need signals that look at the client and its behavior, not just request counts.
Should I block all bots to stop scraping?
Usually not. Search engines, link previews, monitoring, and partner integrations are bots too, and blocking them hurts visibility and operations. Verify the crawlers you want, then block or challenge unverified automation on the routes that matter.

Related posts