The robots.txt file and controlling AI crawlers

The robots.txt file, placed at the root of your site, tells each bot which sections it can or can't crawl — including generative AI crawlers, each identified by its own user-agent (GPTBot, ClaudeBot, PerplexityBot, Google-Extended...) that you can allow or block independently of one another.

What robots.txt does (and doesn't do)

Robots.txt gives crawling instructions, not privacy guarantees or guaranteed indexing control. Three important clarifications:

• Blocking a URL in robots.txt stops a compliant bot from crawling it, but doesn't necessarily stop it from appearing in an index if it's already known through other means (external links, for instance). To actually remove a page from an index, the noindex tag or a removal request through Search Console are the right tools, not robots.txt.
• The file works on good faith: a legitimate, declared bot (Googlebot, GPTBot, ClaudeBot) generally respects the rules it finds there, but nothing technically stops a malicious or poorly coded bot from ignoring them.
• Rules apply per user-agent: you can write different rules for each bot, which is exactly what lets you arbitrate between classic SEO and GEO.

Basic syntax

A robots.txt file is made of blocks, each targeting one or more user-agents:

User-agent: [bot name or * for all]
Disallow: [blocked path]
Allow: [allowed path, takes priority over a broader Disallow]

Simple example blocking an admin folder for all bots, but allowing everything else:

User-agent: *
Disallow: /admin/
Allow: /

Sitemap: https://yourdomain.com/sitemap.xml

The Sitemap: line at the end, while optional, is good practice: it tells bots directly where to find the full list of your pages, independent of any submission made through Search Console or Bing Webmaster Tools.

The main AI user-agents to know

Here are the most common crawlers as of today, keeping in mind this list evolves with the market:

• GPTBot (OpenAI): used to collect content for model training.
• OAI-SearchBot (OpenAI): dedicated to ChatGPT's web search feature, distinct from GPTBot.
• ChatGPT-User (OpenAI): triggers when a user explicitly asks ChatGPT to look at a page in real time.
• ClaudeBot and anthropic-ai (Anthropic): content collection by Anthropic.
• PerplexityBot: crawling for indexing by Perplexity; Perplexity-User handles real-time browsing requests initiated by a user.
• Google-Extended: specifically controls how your content is used by Google's AI models (Gemini), separate from Googlebot, which handles classic indexing.
• Bingbot covers both classic Bing search and, to some extent, the data used by Copilot.

Each of these user-agents can be targeted individually in robots.txt, which lets you, for example, allow Googlebot without allowing Google-Extended, or allow PerplexityBot without allowing GPTBot.

Deciding: allow, block, or differentiate by crawler

The right call depends on your objective, and there's no universal answer:

• Blocking a training crawler (GPTBot, ClaudeBot for collection, Google-Extended) reduces use of your content for training future models, but has no effect on your presence in real-time answers generated by those same assistants, which often go through a separate web search mechanism.
• Blocking a search/citation crawler (OAI-SearchBot, PerplexityBot, Perplexity-User) directly reduces your chances of being cited in those tools' answers, since that's the bot that lets the AI discover and read your page at the moment it answers a question.
• Most sites looking to grow their GEO visibility have an interest in allowing search/citation crawlers, and deciding case by case on training crawlers based on their own IP and content policy.

Example robots.txt that blocks training but allows citation:

User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: *
Allow: /

Sitemap: https://yourdomain.com/sitemap.xml

The case of firewalls and bot management tools

Beyond the robots.txt file itself, some sites use an application firewall or a service like Cloudflare to block or throttle bots at the network level, independent of what's declared in robots.txt. These settings are sometimes turned on by default with broad blocking rules against "bots" in general, which can block a legitimate AI crawler without any trace of it in your robots.txt. If you notice an AI crawler you've explicitly allowed never shows up in your logs, check these network-level settings in addition to the robots.txt file itself: both layers need to line up for your rules to actually produce the intended effect.

Checking and testing your robots.txt

After any change, check the file directly at yourdomain.com/robots.txt in a browser, and use the robots.txt tester built into Google Search Console ("Settings" > "robots.txt" report) to confirm a specific URL is allowed or blocked as intended. A syntax mistake in this file (a Disallow: / left under the wrong user-agent, for instance) can deindex an entire site by accident — treat it with the same care as a production change.

Robots.txt works together with two other files: the XML sitemap and the llms.txt file, which round out the setup by explicitly telling AI systems your pages and your preferences. To go further on the bots themselves, the lesson AI crawlers: who reads you, and why details how to spot them directly in your server logs, beyond what they declare in their user-agent.