AI crawlers have become one of the most important access decisions a publisher makes, and this month the rules around them shifted again. Meta pushed back new business terms that publishers say would allow broad scraping of their sites, Google explained exactly what robots.txt can and cannot control, and fresh data showed that many bots visiting news sites are not who they claim to be. If your site depends on ads, subscriptions or licensing, who gets to read your pages is now a revenue question.
This guide pulls the latest reporting together in plain language. It covers what robots.txt actually does, why impersonation makes simple blocklists unreliable, what changed with Meta’s terms, how AI agents add a new layer, and seven smart ways to manage crawler access without disappearing from search.
Table of Contents
Why AI Crawlers Are Back on the Publisher Agenda
Three stories landed in the same week. On October 15, 2026, Nieman Lab first reported Meta’s new terms for business accounts, which would allow scraping of websites “associated with” business accounts on Facebook, Instagram, WhatsApp and Threads. Virtually every news organization has at least one. Meta later delayed the effective date to November 20.
At the same time, publishers on both sides of the Atlantic pushed for rules that would make bots identify themselves. As Digiday reported, the News/Media Alliance organized more than 300 publishing executives to lobby Congress in September for the Stealth Bot Prohibition Act, and a coalition including The Times, the Financial Times and Condé Nast urged the UK government to back a similar bill on October 15.
The conversation is active on X too. Our count of original public posts mentioning robots.txt alongside AI or crawler terms found 821 posts in the seven days to the morning of October 24, 2026, peaking at 148 on October 18, and posts about Meta and scraping reached 513 in the same window. Google Trends interest in the exact phrase is smaller and steadier, which fits a topic driven by trade press and industry debate rather than consumer search.
What Robots.txt Actually Controls
The most useful clarification this month came from Google. In a Search Central video covered by Search Engine Journal, Google’s Gary Illyes said robots.txt, as defined by the internet standard RFC 9309, is “an access preference protocol.” It governs which URLs a crawler may request, and it “doesn’t dictate what a system does with the data after it has already crawled it.”
That distinction matters for anyone trying to block AI crawlers selectively. Illyes said that to keep one crawler out entirely, you must target its specific user agent token in the file. Invented directives do not work. Google’s robots.txt specification supports four elements: User-Agent, Allow, Disallow and Sitemap. Lines Google does not recognize are ignored.
Google does offer one documented usage control. Google-Extended is a robots.txt token with no crawler of its own. Google crawls with its existing user agents, and the token controls whether that content can be used to train future Gemini models and for grounding in Gemini Apps and Grounding with Google Search on Vertex AI. Google says it does not affect inclusion in Google Search and is not a ranking signal.
A broader fix is still only a proposal. Illyes co-wrote an IETF draft, dated August 19, that would add a Content-Usage rule to robots.txt and a matching HTTP header, so a site could allow crawling while stating a preference such as “train-ai=n”. The draft is still a Working Group document that has not yet gone to the IETF’s steering group for review, and the vocabulary draft that defines the train-ai label says it reflects no consensus yet.
The Fake Bot Problem
Robots.txt only works if a bot is honest about its name. The newest data suggests many are not. Centinel examined crawler identity claims across the news and media sites it protects from late June through September 21. On a typical site, one in seven requests claiming to be Googlebot failed verification. Of requests claiming to be GPTBot, 84% were fake, and 99% of requests claiming to be Perplexity-User were fake.
The fakes are getting more convincing. Centinel found 94% of fake Googlebot requests copied Google’s real crawler version string. A separate Piano analysis of Cloudflare data found that 70% of bot traffic is unverified. DataDome’s September report found scraping accounted for 70.9% of the malicious bot traffic it analyzed, and that scraping rose 185.2% over its 12-month study period.
Large publishers describe the scale in concrete terms. At Digiday’s Media Strategies London event, the Financial Times’ Fiona Spooner said one account alone tried to access about a million articles in eight hours from roughly 700,000 IP addresses. People Inc.’s Jon Roberts said the company now allows through 38 bots a day among “tens of millions” of blocked AI crawler requests.
Roberts was blunt about lists. People Inc. first blocked known AI crawlers and allowed partners through, but some crawlers hid or changed their identity. In January it switched to a default block approach enforced at the CDN. “Robots.txt does nothing,” he said. Reuters also adopted block by default this year, though its digital chief said the approach is “not always 100% foolproof.”
Meta’s Terms and the Robots.txt Question
Meta’s proposed terms show why a voluntary signal leaves publishers uneasy. After the first report, Meta removed the word “train” on October 16 and limited future scraped content to “advertising and business products.” The terms were due to take effect on October 30 with no way to opt out. Meta then moved the date to November 20.
A Meta spokesperson told Nieman Lab that content owners will still be able to use robots.txt to signal preferences, and an earlier statement on October 17 said “News sites, like all other content owners, still have the ability to disable the crawler via robots.txt flags.” But as Nieman Lab noted on October 23, no version of the terms includes language about respecting robots.txt preferences.
Publisher groups pushed back hard. The News/Media Alliance, which represents more than 2,000 US news publishers, wrote to Meta asking for discussions on four core issues, including which crawlers will be used and how robots.txt files, registration walls and paywalls will be respected. WAN-IFRA, whose membership spans 20,000 publishers in 120 countries, sent a similar letter.
For a smaller publisher, the practical lesson is simple. Business tools you use for distribution can come with access terms that reach your website. Read platform term updates the way you would read an ad network contract, and keep your robots.txt and CDN rules ready to express what you allow.
Agents Are the Next Wave
Crawlers that collect pages are only part of the picture. Google’s Sundar Pichai told the Sources podcast that he expects AI agents to reach websites in two ways: by browsing much like a person, and through protocols built for agents. He called protocols “a far more scalable approach” but said they will take longer to develop.
Identification is already a flashpoint for agents. Search Engine Journal noted that Google lists Google-Agent as the user agent for its hosted agents that browse the web. It also reported that Amazon cut off Meta’s Muse agent from shopping on Amazon.com in September, saying the agent did not identify itself when it browsed. Expect the same question publishers ask of crawlers to apply to agents: who are you, and why are you here?
7 Smart Ways to Manage AI Crawlers
The steps below build on the reporting above. They work for a single blog as well as a large newsroom, and they keep the focus on revenue rather than blocking for its own sake.
1. Audit robots.txt by user agent token
List every AI crawler named in your file and what you want for each. Illyes’ point is the starting rule: a crawler is only blocked if its token is targeted explicitly. Remove made up directives, because Google ignores lines outside User-Agent, Allow, Disallow and Sitemap.
2. Separate search from AI training and grounding
Decide on Google-Extended on its own terms. Blocking it does not remove you from Google Search, according to Google, but it does affect Gemini training and grounding. Make that call with your licensing plans in mind rather than by habit.
3. Verify bots that claim a famous name
With one in seven Googlebot claims failing verification in Centinel’s data, a user agent string is not proof. Google publishes guidance for verifying its crawlers, and your CDN or bot management vendor can usually check identity before a request reaches your pages. Treat unverified claims as unknown traffic.
4. Enforce at the edge, not only in a file
Robots.txt states a preference. A CDN or firewall rule enforces one. People Inc. and Reuters both moved to block by default for unidentified bots. If heavy scraping is costing you server load or licensing leverage, put the rules where requests actually arrive.
5. Keep an allowlist and measure it
Know which crawlers you let in and why. People Inc. allows through 38 bots a day. Your list will be shorter, but it should exist in writing, with an owner, and be checked against your server logs every month.
6. Track platform terms that reach your site
Put a calendar reminder on term updates from the platforms you use. Meta’s terms now take effect on November 20, 2026. Confirm what each platform says about crawling, and make sure your robots.txt reflects your choice before any deadline.
7. Connect access decisions to revenue
Every access decision should map to a revenue goal: search traffic, AI referrals or a licensing deal. Our guide to AI content licensing explains how controlled access supports paid deals, and controls you can prove give you leverage at the table.
How to Spot AI Crawlers in Your Own Data
You do not need a bot management contract to start learning who visits your site. Your hosting control panel, CDN dashboard or raw server logs already record the user agent and IP address behind every request. A short monthly review is enough to see patterns that matter for revenue.
- Look for user agents naming AI crawlers, such as GPTBot, Perplexity-User or Googlebot, and check whether those requests come from the operator’s published addresses.
- Watch for many IP addresses requesting long runs of articles in a short time, the pattern the Financial Times described.
- Compare crawler requests with pageviews in your analytics. Requests that never show up as visits still cost server resources.
- Check whether paywalled or registration only pages are being requested by bots you never approved.
- Note sudden spikes after news events or platform term changes, then confirm your rules held.
Write down what you find in a simple sheet: crawler name, verified or not, pages requested, and your decision. Over a few months, that record becomes the evidence you need to tune CDN rules, talk to an ad partner about invalid traffic, or show a licensing partner what access to your archive is actually worth.
Keep one caution in mind. Blocking too aggressively can catch legitimate crawlers, including the search engines that still send most publishers their traffic. Change one rule at a time, keep verified search crawlers allowed, and check your search console reports after any change so a security fix does not become a traffic problem.
What This Means for Publisher Revenue
Managing AI crawlers is no longer just a technical setting. Pew Research Center found 15% of Americans now get news from AI chatbots, up from 9% in 2025, according to Digiday. That audience is reached through the same systems that crawl your pages, so blanket blocking and blanket access both carry costs.
We covered the traffic side of this shift in our look at how AI search is changing publisher traffic. For hands on blocking rules, our earlier guide on how to block LLM bots from scraping ad inventory walks through the technical setup.
The bigger point is control. Publishers want rules that make bots identify themselves, platforms want broad access, and standards bodies are still drafting usage signals. Until those settle, the publishers in the strongest position will be the ones who know exactly which AI crawlers visit, which ones they allow, and what each one is worth.
AI Crawlers FAQ
Can robots.txt stop AI crawlers from training on my content?
Not directly. Google’s Gary Illyes said robots.txt is an access preference protocol that governs which URLs a crawler may request, not what happens to the data afterward. To keep a specific crawler out, you must target its user agent token. A Content-Usage rule for training preferences exists only as an IETF draft.
Does blocking Google-Extended hurt my Google Search rankings?
According to Google, Google-Extended does not affect a site’s inclusion in Google Search and is not a ranking signal. It controls whether crawled content can be used to train future Gemini models and for grounding in Gemini Apps and Grounding with Google Search on Vertex AI.
How many bots fake their identity?
Centinel’s report found that on a typical news site, one in seven requests claiming to be Googlebot failed verification, 84% of requests claiming to be GPTBot were fake, and 99% of requests claiming to be Perplexity-User were fake. A Piano analysis of Cloudflare data found 70% of bot traffic is unverified.
When do Meta’s new business terms take effect?
Meta delayed the effective date of its updated Commercial Terms and Self-Serve Ad Terms from October 30 to November 20, 2026. Meta says content owners can still use robots.txt to signal preferences, but Nieman Lab reported that no version of the terms includes language about respecting robots.txt.
Is robots.txt enough to block scrapers?
Many large publishers say no. People Inc. switched to a default block approach enforced at the CDN in January after crawlers hid or changed their identity, and Reuters also blocks unidentified bots by default. Robots.txt states a preference, while CDN or firewall rules enforce it.
Should publishers block all AI crawlers?
Not automatically. Pew Research Center found 15% of Americans get news from AI chatbots, up from 9% in 2025. Each access decision should map to a goal such as search traffic, AI referrals or a licensing deal, with an allowlist of approved crawlers checked against server logs.
Related posts:
What Is Website Monetization?
How to Monetize AI-Generated Newsletters: 7 Proven Strategies for 2026
10 Ways to Monetize Your Website & Earn Revenue
Snapchat Monetization Requirements: Making Money on Snapchat
The Website Revenue Guide: How Much Money Should Your Site Make?
