Known bots access

WebShield recognizes known bots by their User-Agent and lets you decide, per host, which ones to allow and which to block. This is useful when, for example, you want search engines for indexing but do not want AI crawlers collecting your content for model training.

By default every known bot is allowed. Blocking is opt-in: you uncheck the bots you want to keep out.

Bot groups

Bots are grouped by purpose:

• AI training crawlers — collect content to train models: GPTBot (OpenAI), ClaudeBot and anthropic-ai (Anthropic), CCBot (Common Crawl), Bytespider (TikTok), Amazonbot, Meta-ExternalAgent, Google-CloudVertexBot, Cohere, Diffbot, Timpibot and others.
• AI search crawlers — index for AI search answers: OAI-SearchBot (OpenAI), PerplexityBot.
• AI assistants — fetch a page in response to a user’s chat request: ChatGPT-User, Claude-User, Perplexity-User, Meta-ExternalFetcher, MistralAI-User, DuckAssistBot.
• Search engines — Googlebot, Bingbot, YandexBot, DuckDuckBot, Baiduspider, Applebot, PetalBot and others.
• Social networks & messengers — link previews: Facebook, X (Twitter), LinkedIn, Pinterest, Telegram, Discord, Slack, WhatsApp, VK and others.
• SEO & analysis tools — AhrefsBot, SemrushBot, MJ12bot, DotBot, DataForSeoBot and others.
• Monitoring & uptime — UptimeRobot, Pingdom, StatusCake, Site24x7 and others.

How to configure

1. Open Domains → DNS → Proxy parameters (for a proxied host) or the site card under Sites (for a static one).
2. In the Known bots block, uncheck the bots you want to block. A checked box means “allowed”, unchecked means “blocked”.
3. For a whole group, use Block group / Allow group; to reset everything, use Allow all.
4. Save the changes.

A blocked bot receives a 403 response before reaching your server: no content is served and no load hits your origin.

Common scenarios

• Keep search, block AI: uncheck the “AI training crawlers” and “AI search crawlers” groups while leaving search engines allowed — you keep regular search indexing but stop feeding your content to model training.
• Unique or paid content: block AI crawlers and, if needed, SEO tools that add load and mirror your site structure.
• Monitoring only: for a technical host, you can leave only the monitoring group allowed.

Limits

Recognition is based on the User-Agent. Honest bots from major companies identify themselves with their real name, and blocking applies to them reliably — including bots from verified IP addresses if you deliberately blocked them. However, a User-Agent can be forged: a bot masquerading as a regular browser is not stopped by this layer — other WebShield mechanisms handle those (browser check, signatures, TLS fingerprint, fingerprint and behavior analysis).

Tokens such as Google-Extended and Applebot-Extended control training via robots.txt, not via a separate User-Agent.