Blocking or Allowing AI Crawlers: Copyright and Licensing for Marketers

Blocking AI crawlers is two decisions, not one: whether AI companies may train on your content, and whether AI search may show it. OpenAI and Google document separate controls for each. In the EU, the Copyright Directive lets rightholders reserve text and data mining "by machine-readable means", and the AI Act requires general-purpose model providers to respect that reservation. Most marketers who want AI visibility should allow search crawlers and decide separately about training.

Method: we read Directive (EU) 2019/790 and Regulation (EU) 2024/1689 in the Official Journal, and the crawler pages from OpenAI and Google, on 31 October 2026 (CET). Quotes are verbatim. This isn't legal advice, and opinions are marked "Our view".

On this page

• What does EU copyright law say about AI training?
• How does the AI Act connect to that opt-out?
• Which crawler controls what?
• What should a marketer actually choose?
• Is robots.txt the only way to opt out?
• What about licensing deals?
• What about outside the EU?
• Checklist
• From MentionX data: what you give up by blocking search crawlers
• Worked example: Ledgerline sets its crawler policy
• Who should make the decision?
• How do you check the policy works?
• What about content you've licensed from others?
• What does blocking not do?
• How often should you review?
• Should small businesses worry about this?
• How do licensing deals fit in?
• What should you tell your team?
• What about content outside the EU?
• Common questions
• Sources

Checked on 31 October 2026. "Our view" marks MentionX opinion. Ledgerline is a fictional company used for illustration.

What does EU copyright law say about AI training?

Article 4 of the Copyright in the Digital Single Market Directive creates an exception for "reproductions and extractions of lawfully accessible works and other subject matter for the purposes of text and data mining" (Directive 2019/790, Art. 4(1)).

The exception has a condition. It applies "on condition that the use of works and other subject matter referred to in that paragraph has not been expressly reserved by their rightholders in an appropriate manner, such as machine-readable means in the case of content made publicly available online" (Directive 2019/790, Art. 4(3)).

In plain terms: by default, mining publicly accessible content is allowed under this exception. You can opt out, and for online content the Directive names machine-readable means as an example of how.

Copies made under the exception "may be retained for as long as is necessary for the purposes of text and data mining" (Directive 2019/790, Art. 4(2)).

How does the AI Act connect to that opt-out?

The AI Act requires providers of general-purpose AI models to "put in place a policy to comply with Union law on copyright and related rights, and in particular to identify and comply with, including through state-of-the-art technologies, a reservation of rights expressed pursuant to Article 4(3) of Directive (EU) 2019/790" (EU AI Act, Art. 53(1)(c)).

They must also "draw up and make publicly available a sufficiently detailed summary about the content used for training" (EU AI Act, Art. 53(1)(d)). Chapter V, where Article 53 sits, has applied since 2 August 2025 (EU AI Act, Art. 113).

Our view: this makes your opt-out signal matter more, because model providers must have a policy for detecting it. Whether a given robots.txt line counts as an "appropriate manner" in a dispute is for courts, not this guide.

Which crawler controls what?

Each company documents its own user agents. The key distinction is training versus search.

OpenAI recommends "allowing OAI-SearchBot in your site's robots.txt file and allowing requests from our published IP ranges" if you want to appear in search results (OpenAI).

Google also says it "intends to support the AIPREF shared machine-readable training preference signal (with "user-agent: __usage_ai-training") in early 2027" (Google). Watch for that.

For Anthropic and Perplexity, read their crawler pages directly: (Anthropic) and (Perplexity). We haven't summarised them here because their user-agent lists change.

What should a marketer actually choose?

Our view: decide by content type and goal, not by one site-wide rule.

Note that Google-Extended also covers grounding in Gemini Apps, so blocking it may reduce how Gemini uses your content, though Google says it doesn't affect Search.

Fictional example: a B2B software company wants its product pages in ChatGPT answers but doesn't want its paid research reports used for training. It allows OAI-SearchBot site-wide and disallows GPTBot and Google-Extended only on /research/.

User-agent: GPTBot
Disallow: /research/

User-agent: Google-Extended
Disallow: /research/

User-agent: OAI-SearchBot
Allow: /

Is robots.txt the only way to opt out?

No. The Directive's example is "machine-readable means", and robots.txt is the most widely documented one. Our view: also state the reservation in your terms of use, so you have both, and keep a dated copy of your robots.txt so you can show when the reservation existed.

Network-level tools are another option. Cloudflare documents blocking AI bots and a pay-per-crawl feature (Cloudflare; Cloudflare pay per crawl). Those enforce access, rather than only stating a preference.

What about licensing deals?

If you own valuable content, an opt-out gives you something to negotiate with: a model provider that wants it has to ask. Smaller publishers rarely get direct deals, so our view is that for most brands, visibility is worth more than holding out.

What about outside the EU?

This guide covers EU law. In the US, the Copyright Office has published a series of reports on AI and copyright (US Copyright Office). Other jurisdictions differ, so check locally.

Checklist

1. Separate training bots from search bots in robots.txt.
2. Allow the search bots for every page you want cited.
3. Decide on training per section, not site-wide.
4. Keep dated copies of robots.txt and your terms.
5. Recheck each vendor's crawler page quarterly. See also the EU AI Act for marketers.

From MentionX data: what you give up by blocking search crawlers

Method. Read-only aggregate queries over MentionX scans from 22 August to 29 October 2026 (UTC): 2,217 engine runs (2,154 completed) across 18 anonymised projects and six engines.

Blocking a search crawler can remove you from that assistant's cited answers. Our sample shows how much citation activity varies by engine:

And which kinds of pages are cited:

Our view: product, documentation and comparison pages are commonly cited, so blocking AI search crawlers on those paths gives up visibility. Training opt-outs are a separate choice and don't, by vendors' own documentation, affect search inclusion in the same way.

Limits. A convenience sample of mostly B2B software projects, run through engine APIs, which can behave differently from the consumer apps. Perplexity, Copilot and DeepSeek figures come from a single day (26 October 2026), so treat them as a snapshot, not a trend.

Worked example: Ledgerline sets its crawler policy

Ledgerline is a fictional accounts-payable tool. Figures are illustrative.

Ledgerline publishes free guides, product pages and a paid benchmark report. Legal asks marketing to protect the paid report without losing AI visibility.

The team allows search crawlers site-wide, disallows training crawlers on the report path, and puts the report behind a login with paywall markup. It records the decision, the date and the user agents covered in a short policy note, and sets a quarterly reminder to check vendors' crawler pages for new user agents.

Who should make the decision?

Crawler policy touches marketing, legal and engineering. Marketing knows which pages drive visibility. Legal knows licensing obligations and risk. Engineering implements robots.txt and server rules. Agree a short written policy that names an owner and says how changes are approved. Our view: the marketing owner should propose, legal should sign off on anything involving paid or licensed content, and engineering should implement and test.

How do you check the policy works?

After changing robots.txt, test it with a robots.txt tester and fetch a few blocked and allowed URLs as each user agent would. Then check server logs over the following weeks: blocked agents should stop fetching the blocked paths. Note that robots.txt is a request, not a technical barrier. If content must not be fetched, put it behind authentication.

What about content you've licensed from others?

If you publish content under a licence, such as stock images, syndicated articles or partner data, read the licence terms. Some restrict use by third parties, including AI systems. You may need to block crawlers on those paths, or ask the licensor what they require. When in doubt, ask legal.

What does blocking not do?

Blocking a crawler today doesn't remove content already collected. It also doesn't stop third-party sites quoting or summarising your pages, and those pages may be crawled. Some assistants fetch pages live when a user asks about them, using different user agents from their training crawlers; check each vendor's documentation for which agent does what.

How often should you review?

Quarterly is a sensible rhythm. Vendors add and rename user agents, and new signals for training preferences are being developed. Check each vendor's crawler page, update robots.txt and record the review date. Review sooner when a major vendor announces a change.

Should small businesses worry about this?

Most small businesses gain more from being visible in AI answers than they lose from training use, because their pages are marketing pages. The main cases for blocking are paid content, licensed material and content your business model depends on keeping exclusive. If none apply, allowing crawlers is usually the simpler choice.

How do licensing deals fit in?

Some large publishers have signed licensing agreements with AI companies. These are negotiated individually and are rarely an option for most businesses. If you hold valuable content, such as research, data or archives, it may be worth asking legal whether licensing is realistic. For most marketing teams, the practical choice is between allowing and blocking crawlers.

What should you tell your team?

Share a one-page summary: which crawlers are allowed, which paths are blocked, why, and who to ask. Writers and product managers should know that new paid content needs a robots.txt update before launch. Add a line to your launch checklist so it isn't missed. When a vendor adds a new user agent, the owner updates the summary and robots.txt together, so the written policy and the live file always match.

What about content outside the EU?

Copyright rules on AI training differ by country and are still being argued in courts. If you publish for several markets, set one policy that works for your strictest market, then ask legal whether any market needs more.

Common questions

Does blocking GPTBot remove me from ChatGPT search? OpenAI documents GPTBot as its training crawler and OAI-SearchBot for search. Check its crawler page for current details.

Does Google-Extended affect Google Search? Google says it doesn't affect inclusion in Search or ranking.

Is robots.txt legally binding? That's a legal question. It's a widely used machine-readable signal; ask legal about your situation.

Sources

All accessed 31 Oct 2026.

• European Union, Directive (EU) 2019/790 (Copyright in the DSM) (Official Journal, 17 May 2019)
• European Union, Regulation (EU) 2024/1689 (AI Act) (Official Journal, 12 Jul 2024)
• OpenAI, Overview of OpenAI crawlers
• Google, Google's common crawlers
• Anthropic, Crawler help article; Perplexity, Perplexity crawlers
• Cloudflare, Block AI bots; Pay per crawl
• US Copyright Office, Copyright and AI