Allow trusted crawlers. Control the rest.
For news and media: Allow Googlebot and other search crawlers; set AI crawlers such as GPTBot to Require payment or Block. Compare crawls with referrals first.
Trusted by global brands, public institutions, and universities
Crawler value management
Decide which crawlers create value then Allow, Require payment, or Block.
- Value vs non-value crawlers
- See which crawlers return referrals and which only consume load.
- Crawls vs referrals
- Compare crawl volume with traffic sent back before you set a rule.
- Allow, Require payment, or Block
- One rule per crawler. Allow search or licensed free crawl; Require payment when crawl should not be free; Block the rest.
- Protect infrastructure
- Cut low-value crawl load while keeping high-value access.
Named AI crawlers
Control GPTBot and other AI crawlers without touching search bots.
Allow, Require payment, or Block — per crawler, without touching search bots.
- GPTBot
- OpenAI training crawler.
- ClaudeBot
- Anthropic training crawler.
- Bytespider
- ByteDance AI crawler.
- PerplexityBot
- Perplexity AI search crawler.
- Google-Extended
- Google AI use. Keep Googlebot; control this separately.
- And many more
- Set Allow, Require payment, or Block for other known AI crawlers the same way.
Crawler Analysis
See which crawlers create value from crawl and referral data.
- Detect crawler traffic.
- Label search vs AI crawlers in real time.
- Measure referrals.
- Compare crawl cost with traffic sent back.
Define who is allowed to crawl
Set access rules per crawler. Changes save automatically.
| Crawler | Type | Access rule |
|---|---|---|
Googlebot SEO indexing | Search Engines | |
Bingbot Search indexing | Search Engines | |
GPTBot Blocked | AI Services | |
ClaudeBot Payment required | AI Services | |
Bytespider Blocked | AI Services |
Click an access rule to change it.
Crawler Management
Allow, Require payment, or Block per crawler.
- Enforce per crawler.
- Allow, Require payment, or Block for GPTBot, ClaudeBot, Bytespider, and more.
- Require payment.
- API returns require_payment — route to any payment collector, or allow if already paid. Not an Atmosvere checkout.
- Match license deals.
- Allow or Require payment under contract.
- Protect origin.
- Stop unwanted crawlers from hammering your servers.
Key benefits
Keep SEO. Control AI crawlers.
- SEO-safe allowlisting.
- Allow Googlebot and other search crawlers; set AI crawlers apart.
- Licensed AI access.
- Under contract, Allow free crawl — or Require payment when access should not be free. Monthly budgets can escalate to Require payment.
- Spoof detection.
- Catch clients that fake a known crawler identity.
- Infrastructure protection.
- Block heavy crawlers so origin stays free for readers.
Frequently asked questions
- Will crawler management hurt SEO?
- No. Keep Googlebot on Allow. Set AI crawlers such as GPTBot to Require payment or Block separately.
- What are Allow, Require payment, and Block?
- Allow lets the crawler through. Require payment returns a payment-required decision in the API instead of a hard block. Block stops the crawler. Most publishers use Allow and Block; monthly budgets under a license can escalate to Require payment.
- What does Require payment mean?
- The API returns require_payment instead of block. You can send the crawler to any third-party payment collector, or let it through if it has already paid. Atmosvere does not run checkout. Use Block when you want a hard stop with no payment path.
- Can we block GPTBot and still allow Googlebot?
- Yes. Allow Googlebot; set GPTBot, ClaudeBot, Bytespider, PerplexityBot, or Google-Extended to Block, Require payment, or Allow if you have a license deal.
Testimonials
Trusted by teams across Europe who rely on Bot Shield to protect applications, APIs, and infrastructure.
“Atmosvere was easy to integrate, simple to run, and the team behind it provides highly responsive, personal support whenever needed.”
“Atmosvere gave us full control over automated traffic. We reduced unnecessary load, blocked bots, managed crawlers, and improved system stability—while also strengthening our marketing activities without impacting real users.”
“Anyone who uses Atmosvere is sending a clear signal about a sustainable and digitally ethical platform economy.”
“Atmosvere is really easy to implement and has been evolving quickly - the team is always open in providing new insights and turning feedback into real improvements.”
European by design
Built for European trust. Austrian company. European-owned infrastructure. GDPR-first by default.
- Requests analyzed
- Billions
- Bot checks
- Hundreds
- Infrastructure
- EU-owned
- Mission
- 1