Dashboard
EN
Start free
Updated

Meta's updated Commercial Terms and crawlers: user-agents, logs and robots.txt

What is changing

Meta is updating its Meta Commercial Terms, which apply to access or use of Meta products for a business or commercial purpose. Meta lists examples such as using ads, selling products, developing apps and managing a Page. According to Meta’s official terms page, the update takes effect on November 20, 2026.

What the press reported

Nieman Lab and Search Engine Journal reported that Meta delayed the update (earlier coverage cited October 30, 2026) and that the new text would let Meta crawl websites linked to businesses that use its products. Nieman Lab reported that Meta said site owners can still use robots.txt to signal preferences, and that legal experts remain skeptical. We summarize these reports here; read the official terms for the exact wording. This page is not legal advice.

The current Commercial Terms also say that continuing business use of Meta products after notice of an update means you agree to the updated terms.

Meta crawlers you may see in your logs

From Meta’s official Web Crawlers documentation (page updated May 21, 2026):

Crawler Purpose (per Meta) User-agent contains robots.txt (per Meta)
FacebookExternalHit Link previews when content is shared on Facebook, Instagram, Messenger facebookexternalhit/1.1 Might bypass it for security or integrity checks
Meta-ExternalAgent Use cases such as training foundation AI models or improving products by indexing content meta-externalagent/1.1 Configurable via robots.txt; no bypass exception documented
Meta-ExternalFetcher Fetches individual links at a user’s request, including agentic AI tasks meta-externalfetcher/1.1 May bypass it (user-requested fetches)
Meta-WebIndexer Improving Meta AI search results and citations meta-webindexer/1.1 Configurable via robots.txt
Meta-ExternalAds Improving advertising and other business products meta-externalads/1.1 Configurable via robots.txt

Full strings may include a suffix, for example facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php) or meta-externalagent/1.1 (+/documentation/sharing/webmasters/web-crawlers).

Quick log check (nginx/Apache combined log):

grep -iE "facebookexternalhit|meta-external(agent|fetcher|ads)|meta-webindexer" access.log \
  | awk '{print $1, $7}' | sort | uniq -c | sort -rn | head

User-agent strings can be spoofed. In its FacebookExternalHit requirements, Meta calls allowlisting by IP address more secure than by user-agent, so check source IPs before you trust a match.

How robots.txt applies

Block one Meta crawler:

User-agent: meta-externalagent
Disallow: /

Allow everything except one directory (Meta’s own example):

User-agent: meta-externalagent
Allow: /
Disallow: /private/

Meta says to allow up to 24 hours for changes, because its crawlers may cache robots.txt for that long.

Collecting web data for your own agents

If you need structured web data for your own agents or pipelines, you don’t have to run a crawler fleet. FastAF offers scrape, SERP and other APIs with pay per call via x402. See pricing.

FAQ

When do Meta’s updated Commercial Terms take effect?

November 20, 2026, according to Meta’s official Commercial Terms page.

Does Meta respect robots.txt?

Meta’s documentation explains how to set robots.txt rules for its crawlers. It says Meta-ExternalFetcher may bypass robots.txt because it fetches links a user requested, and FacebookExternalHit might bypass it for security or integrity checks.

Which user-agent does Meta use for AI training?

Meta-ExternalAgent, with user-agent meta-externalagent/1.1. Meta describes its use cases as training foundation AI models or improving products by indexing content.

How long does a robots.txt change take to apply to Meta crawlers?

Up to 24 hours, because Meta’s crawlers may cache robots.txt for up to 24 hours.

Do the updated terms apply if I don’t advertise on Meta?

Meta’s Commercial Terms cover business or commercial use of Meta products, which Meta says includes things like selling products, developing apps or managing a Page. Check the official text for your situation; this page is not legal advice.

Sources