Updated Oct 28, 2026

# Meta's updated Commercial Terms and crawlers: user-agents, logs and robots.txt

## What is changing

Meta is updating its **Meta Commercial Terms**, which apply to access or use of Meta products for a business or commercial purpose. Meta lists examples such as using ads, selling products, developing apps and managing a Page. According to Meta’s official terms page, the update takes effect on **November 20, 2026**.

### What the press reported

Nieman Lab and Search Engine Journal reported that Meta delayed the update (earlier coverage cited October 30, 2026) and that the new text would let Meta crawl websites linked to businesses that use its products. Nieman Lab reported that Meta said site owners can still use robots.txt to signal preferences, and that legal experts remain skeptical. We summarize these reports here; read the official terms for the exact wording. This page is not legal advice.

The current Commercial Terms also say that continuing business use of Meta products after notice of an update means you agree to the updated terms.

## Meta crawlers you may see in your logs

From Meta’s official Web Crawlers documentation (page updated May 21, 2026):

Crawler

Purpose (per Meta)

User-agent contains

robots.txt (per Meta)

FacebookExternalHit

Link previews when content is shared on Facebook, Instagram, Messenger

`facebookexternalhit/1.1`

Might bypass it for security or integrity checks

Meta-ExternalAgent

Use cases such as training foundation AI models or improving products by indexing content

`meta-externalagent/1.1`

Configurable via robots.txt; no bypass exception documented

Meta-ExternalFetcher

Fetches individual links at a user’s request, including agentic AI tasks

`meta-externalfetcher/1.1`

May bypass it (user-requested fetches)

Meta-WebIndexer

Improving Meta AI search results and citations

`meta-webindexer/1.1`

Configurable via robots.txt

Meta-ExternalAds

Improving advertising and other business products

`meta-externalads/1.1`

Configurable via robots.txt

Full strings may include a suffix, for example `facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)` or `meta-externalagent/1.1 (+/documentation/sharing/webmasters/web-crawlers)`.

Quick log check (nginx/Apache combined log):

```
grep -iE "facebookexternalhit|meta-external(agent|fetcher|ads)|meta-webindexer" access.log \
  | awk '{print $1, $7}' | sort | uniq -c | sort -rn | head
```

User-agent strings can be spoofed. In its FacebookExternalHit requirements, Meta calls allowlisting by IP address more secure than by user-agent, so check source IPs before you trust a match.

## How robots.txt applies

Block one Meta crawler:

```
User-agent: meta-externalagent
Disallow: /
```

Allow everything except one directory (Meta’s own example):

```
User-agent: meta-externalagent
Allow: /
Disallow: /private/
```

Meta says to allow **up to 24 hours** for changes, because its crawlers may cache robots.txt for that long.

## Collecting web data for your own agents

If you need structured web data for your own agents or pipelines, you don’t have to run a crawler fleet. FastAF offers [scrape, SERP and other APIs](https://fastaf.co/apis/) with pay per call via [x402](https://fastaf.co/x402/). See [pricing](https://fastaf.co/pricing/).

## FAQ

### When do Meta’s updated Commercial Terms take effect?

November 20, 2026, according to Meta’s official Commercial Terms page.

### Does Meta respect robots.txt?

Meta’s documentation explains how to set robots.txt rules for its crawlers. It says Meta-ExternalFetcher may bypass robots.txt because it fetches links a user requested, and FacebookExternalHit might bypass it for security or integrity checks.

### Which user-agent does Meta use for AI training?

Meta-ExternalAgent, with user-agent `meta-externalagent/1.1`. Meta describes its use cases as training foundation AI models or improving products by indexing content.

### How long does a robots.txt change take to apply to Meta crawlers?

Up to 24 hours, because Meta’s crawlers may cache robots.txt for up to 24 hours.

### Do the updated terms apply if I don’t advertise on Meta?

Meta’s Commercial Terms cover business or commercial use of Meta products, which Meta says includes things like selling products, developing apps or managing a Page. Check the official text for your situation; this page is not legal advice.

## Sources

*   Official: Meta Commercial Terms: [https://www.facebook.com/legal/commercial\_terms](https://www.facebook.com/legal/commercial_terms)
*   Official: Meta Web Crawlers documentation: [https://developers.facebook.com/docs/sharing/webmasters/web-crawlers](https://developers.facebook.com/docs/sharing/webmasters/web-crawlers)
*   Press: Nieman Lab: [https://www.niemanlab.org/2026/10/meta-delays-rollout-of-terms-that-would-allow-broad-scraping-of-news-sites-legal-experts-are-still-skeptical/](https://www.niemanlab.org/2026/10/meta-delays-rollout-of-terms-that-would-allow-broad-scraping-of-news-sites-legal-experts-are-still-skeptical/)
*   Press: Search Engine Journal: [https://www.searchenginejournal.com/meta-delays-business-terms-crawling/594388/](https://www.searchenginejournal.com/meta-delays-business-terms-crawling/594388/)