Guides: web crawlers, robots.txt and platform terms

Which crawlers visit your site, how to find them in your access logs and what robots.txt lets you control, read from the official documentation and dated.

• Meta's updated Commercial Terms and crawlers: user-agents, logs and robots.txt

  Meta's updated Commercial Terms take effect Nov 20, 2026. See Meta's crawler user-agents, how to find them in your logs and how robots.txt applies.

Looking for step-by-step guides to read public data from a specific source? See thehow-to guides.