Scraping Google Search without getting blocked: what actually happens

“Google blocks scrapers” is true but too vague to act on. Here is what actually happens, in order, when a script starts sending it automated queries, and why the usual first fix (more proxies) only delays the problem.

Rate limiting is per IP, and it is stricter than you think

Google throttles based on request volume from a single IP address in a short window. Datacenter IP ranges get a noticeably tighter limit than residential ones, because Google can tell a request is coming from AWS, Google Cloud, or a hosting provider rather than someone’s home connection. A handful of rapid requests from one datacenter IP is often enough to trigger a challenge. This is the first wall almost everyone hits, and it is why “just add a delay” works for exactly as long as your request volume stays trivial.

CAPTCHA and the consent wall

Past the rate limit, Google’s next response is not an error, it is a different page: a CAPTCHA challenge, or in EU/UK traffic, a cookie consent interstitial that has to be accepted before search results render at all. Both break a script that expects organic_results at the top of the HTML. Neither is solvable by retrying: a CAPTCHA needs a human (or a paid solving service, which is its own cost and its own reliability problem), and a consent wall needs a session that has already accepted it, which means cookie handling most scrapers do not implement.

Fingerprinting goes beyond the IP

IP address is the first signal, not the only one. Google’s anti-automation systems also look at TLS handshake characteristics, HTTP header order and completeness, browser-level signals when a headless browser is involved (viewport size, installed fonts, WebGL renderer string, whether navigator.webdriver is set), and behavioral patterns like requests arriving at suspiciously regular intervals. A request that clears the rate limit but has a bare User-Agent header and none of the headers a real Chrome request would send still reads as automated. This is why swapping proxies without also fixing the rest of the request fingerprint buys very little.

The pagination math nobody budgets for

Google no longer honors the num=100 parameter that used to return 100 results in a single request. Getting “the top 100 results” for a query now means ten separate paginated requests, ten results per page. If your rank-tracking spreadsheet was built around the old assumption, your real request volume, and your real exposure to the rate limit and fingerprinting problems above, is 10x what the spreadsheet says. This alone has broken cost estimates and infrastructure plans written before the change and never revisited.

Why proxies alone do not solve it

Rotating residential proxies fix exactly one of the four problems above: the IP-based rate limit. They do nothing for CAPTCHA pages that already triggered, nothing for TLS and header fingerprinting, and nothing for the pagination math. A proxy pool paired with a naive requests.get and default headers will still get challenged, just from a rotating set of IPs instead of one. Solving all four problems yourself means running a real browser engine (for TLS and JS fingerprint parity), maintaining header profiles that match real traffic, handling consent flows, and paying for proxy bandwidth, continuously, as Google’s detection changes. That is a standing infrastructure commitment, not a one-time script.

The API route

The alternative is to not run any of that yourself:

import requests

API_KEY = "faf_live_..."  # your FastAF key

resp = requests.get(
    "https://api.fastaf.co/v1/serp",
    params={"query": "web scraping api", "country": "us"},
    headers={"Authorization": f"Bearer {API_KEY}"},
    timeout=60,
)
resp.raise_for_status()

for result in resp.json()["organic_results"]:
    print(result["rank"], result["title"], result["link"])

The request goes through infrastructure built to handle rate limiting, fingerprinting and consent walls, and you get back parsed JSON: organic_results, and people_also_ask, knowledge_graph, ads or related_searches when the query produces them. A search costs 5 credits, and failed requests are never billed, since credits are reserved when the request starts and returned if no result comes back.

The short version

Four separate mechanisms stack up against a raw scraper: IP rate limits, CAPTCHA and consent walls, fingerprinting beyond the IP, and a pagination cost that is 10x what it looks like on paper. A proxy pool addresses one of the four. Whether that is worth building depends on how much of your engineering time you want to spend on search infrastructure instead of on the product that consumes the results. See pricing for the credit cost at your volume.