Dashboard
EN
Start free

Why your AI agent’s fetch tool comes back with an empty page

Empty shell, cookie wall, login redirect or a page cut short: how to tell them apart in 30 lines of Python, with real measurements from 19 October 2026.

“I couldn’t read that page” is one sentence from the model and at least five different problems underneath. The fix depends entirely on which one you hit, so diagnose first.

Why does an agent’s fetch tool see nothing useful?

Most built-in fetch tools do the same thing: one plain HTTP GET, then HTML converted to text, then a length cap. No JavaScript runs, no cookies are kept, nobody clicks “accept”. Anthropic’s own documentation for its web fetch tool says, as of October 2026, that it does not support websites dynamically rendered with JavaScript, and that content over max_content_tokens is truncated.

So five situations collapse into the same “nothing useful” for the model:

Case What the server sent Signal you can check
Client-rendered app An HTML shell, content built later by scripts Under 500 chars of visible text, <noscript> asking for JavaScript, or HTML bytes far larger than text
Consent wall A cookie or privacy interstitial Redirect to a consent. host, short page full of “cookie” and “consent”
Login wall A sign-in form Redirect to /login or /signin, a password input
Blocked An error or a challenge 401, 403, 429 or 503
Too long The full page Visible text larger than the tool’s budget

For why rendering costs more and how to spot it by hand, see JavaScript rendering in web scraping. This post is about doing the check in code, inside the agent.

A diagnostic function you can run

Plain requests and BeautifulSoup, no API key, no cost:

import re
import requests
from bs4 import BeautifulSoup

HEADERS = {"User-Agent": "page-diagnostic/1.0", "Accept": "text/html"}
CONSENT = re.compile(r"cookie|consent|gdpr|privacy choices", re.I)
LOGIN = re.compile(r"/(login|log-in|signin|sign-in|auth)\b", re.I)

def diagnose(url, budget=12_000):
    r = requests.get(url, headers=HEADERS, timeout=30)
    soup = BeautifulSoup(r.text, "html.parser")
    noscript = " ".join(n.get_text(" ", strip=True) for n in soup.find_all("noscript"))
    for tag in soup(["script", "style", "noscript", "template"]):
        tag.decompose()
    text = soup.get_text(" ", strip=True)

    if r.status_code in (401, 403, 429, 503):
        verdict = "blocked"
    elif LOGIN.search(r.url) or soup.find("input", {"type": "password"}):
        verdict = "login_wall"
    elif "consent" in r.url or (len(text) < 3_000 and CONSENT.search(text)):
        verdict = "consent_wall"
    elif len(text) < 500 and ("javascript" in noscript.lower() or len(r.content) > 50 * len(text)):
        verdict = "client_rendered"
    elif len(text) > budget:
        verdict = "too_long"
    else:
        verdict = "ok"
    return {"status": r.status_code, "final_url": r.url, "html_bytes": len(r.content),
            "text_chars": len(text), "verdict": verdict, "text": text}

The thresholds are heuristics. Tune them on your own traffic, but the order matters: check the cheap, unambiguous signals (status, URL) before the fuzzy ones (text length).

What it reported on four public pages

Run on 19 October 2026 from a machine in Italy:

URL Status HTML bytes Visible text Verdict
python.org/about/ 200 41,239 4,598 chars ok
crates.io/crates/serde 200 5,406 32 chars client_rendered
it.yahoo.com 200, redirected to consent.yahoo.com 112,226 2,471 chars consent_wall
en.wikipedia.org/wiki/World_War_II 200 2,074,721 196,628 chars too_long

Every one returned 200. A status-code check alone would have called all four a success. crates.io sends a 5 KB shell whose only text is the title and a <noscript> line asking for JavaScript. Yahoo’s Italian homepage redirects a cookie-less visitor to a consent page; uk.yahoo.com, fetched the same way, did not.

What do you do with each verdict?

ok: use the plain fetch. Python.org cost nothing and needed nothing else.

client_rendered: render it. FastAF’s web scraping API does this with dynamic=true for 5 credits instead of 1, or render_js: true on the scrape_page MCP tool. On crates.io a static call returned 0 characters of markdown; a rendered call returned 4,939. Be honest with yourself about variance, though: we made six rendered calls to that page on 19 October, four came back with 2,313 to 4,939 characters and two came back empty, because the app had not finished loading. Those two were charged at the time; after measuring it for this post we changed the rule, and an empty result now returns 422 target_unavailable and costs nothing. A render can still return a thin shell, so check the text length, and set wait (0 to 35,000 ms) for slow apps.

consent_wall: rendering does not help. FastAF’s fetch of it.yahoo.com also got the consent page (3,697 characters, all privacy notice). Nobody should accept terms on a site’s behalf in an automated run. Report it, or get the information from a source designed for it, such as a news search.

login_wall and blocked: stop and say so. Content behind a login is not public, and bot protection is the site owner’s decision. FastAF only serves public pages: a request for https://example.com/login returns 400 blocked_target in 64 ms and costs nothing. The rules are in the acceptable use policy.

too_long: ask for markdown with a budget. The MCP scrape_page tool on the Wikipedia page returned the first 12,000 of 597,765 markdown characters with truncated: true, for 1 credit in 1.3 seconds. The flag is the point: the model knows it read a slice, not the whole page.

A fetch strategy for agents

Cheap path first, rendering only when the diagnostic asks for it, and a hard cap per run:

import os, time, requests

def read_page(url, state, max_chars=12_000):
    d = diagnose(url, budget=max_chars)
    if d["verdict"] in ("blocked", "login_wall", "consent_wall"):
        return {"error": f"{d['verdict']}: page not readable, tell the user"}
    if d["verdict"] in ("ok", "too_long"):
        t = d["text"]
        return {"text": t[:max_chars], "truncated": len(t) > max_chars}
    if state["renders_left"] <= 0:
        return {"error": "page needs rendering, render budget for this run is spent"}
    for _ in range(3):
        r = requests.get("https://api.fastaf.co/v1/scrape",
            params={"url": url, "formats": "markdown", "dynamic": "true", "wait": 3000},
            headers={"Authorization": f"Bearer {os.environ['FASTAF_API_KEY']}"}, timeout=90)
        if r.status_code != 429:
            break
        time.sleep(int(r.headers.get("Retry-After", 3)))
    if r.status_code != 200:
        return {"error": r.json().get("error", {}).get("code", "scrape_failed")}
    state["renders_left"] -= 1
    if len(r.text) < 500:
        return {"error": "rendered page still empty, tell the user"}
    return {"text": r.text[:max_chars], "truncated": len(r.text) > max_chars}

Our final run on 19 October, with renders_left set to 1, cost 5 credits in total. Python.org and Wikipedia came from the free plain fetch (Wikipedia cut to 12,000 of 196,628 characters and flagged), crates.io/crates/serde rendered to 2,508 characters in 6.3 seconds, crates.io/crates/tokio got “budget spent” and Yahoo got consent_wall. Errors come back as tool results so the model can tell the user what happened instead of guessing. Full parameters are in the docs.

The short version

An empty fetch is a symptom, not a diagnosis. Check status, redirect target, visible text length and <noscript> before spending anything. Plain fetch for static pages, rendering at 5 credits only for client-rendered public pages, a per-run render cap, and an honest “I can’t read this” for consent walls, logins and blocks. Truncated pages need a budget and a flag, not a bigger context. Get 500 free credits and try the diagnostic on the pages your agent keeps failing on.