How to scrape Google search results in Python

Every scraping project that touches Google starts the same way: requests.get("https://www.google.com/search?q=..."), a bit of BeautifulSoup, and a parser that works beautifully for about two days.

Then it stops. Here is why it stops, and what a setup that keeps working actually looks like.

Why the DIY version breaks

Four separate things break it, usually in this order:

1. IP blocking. Datacenter IPs get rate limited quickly, then served a consent page or a CAPTCHA instead of results. Rotating a handful of proxies delays this, it does not solve it.
2. Markup churn. Google’s result markup carries obfuscated, rotating class names. A CSS selector that matches today is not a stable contract.
3. Result types. Organic links are the easy part. Featured snippets, People Also Ask, knowledge panels, ads and local packs each have their own structure, and their presence varies by query and country.
4. Pagination maths. Google no longer honours a num=100 parameter, so “the top 100 results” means ten separate paginated requests, not one. Any cost estimate written before that change is wrong by 10x.

None of these are hard problems individually. Together they are a maintenance treadmill: you end up owning a proxy pool and a parser that needs attention every quarter, to get data that is not your product.

The version that keeps working

Send the query to an API that owns the proxies and the parsing, and get JSON back:

import requests

API_KEY = "faf_live_..."

def search(query, country="us", page=0):
    response = requests.get(
        "https://api.fastaf.co/v1/serp",
        params={"query": query, "country": country, "page": page},
        headers={"Authorization": f"Bearer {API_KEY}"},
        timeout=60,
    )
    response.raise_for_status()
    return response.json()

data = search("web scraping api")
for result in data["organic_results"]:
    print(result["rank"], result["title"], result["link"])

No proxy configuration, no selectors. The response also carries people_also_ask, knowledge_graph, ads and related_searches when the query produces them, so you check for a key instead of writing a parser per result type.

Country and language targeting

Rank data is meaningless without a location. Pass the country explicitly rather than hoping the request lands somewhere useful:

for country in ["us", "gb", "de", "it"]:
    data = search("project management software", country=country)
    top = data["organic_results"][0]
    print(country, "->", top["link"])

The query is issued from that location, with the matching Google domain and language defaults.

Collecting more than one page

Since results come ten to a page, treat depth as a loop and remember that each page is a separate billable request:

def search_deep(query, pages=3, country="us"):
    results = []
    for page in range(pages):
        data = search(query, country=country, page=page)
        results.extend(data.get("organic_results", []))
    return results

top_30 = search_deep("headless browser", pages=3)
print(len(top_30), "results")

Handling errors properly

Three status codes matter in production, and treating them all as “retry” is how you burn a credit balance in an afternoon:

import time

def search_safe(query, **kwargs):
    for attempt in range(3):
        response = requests.get(
            "https://api.fastaf.co/v1/serp",
            params={"query": query, **kwargs},
            headers={"Authorization": f"Bearer {API_KEY}"},
            timeout=60,
        )

        if response.status_code == 200:
            return response.json()

        # Out of credits: retrying will never help.
        if response.status_code == 402:
            raise RuntimeError("Out of credits, top up before retrying")

        # Concurrency limit: wait exactly as long as the API asks.
        if response.status_code == 429:
            time.sleep(int(response.headers.get("Retry-After", 1)))
            continue

        # Upstream failure: not billed, safe to retry with backoff.
        if response.status_code in (502, 503, 504):
            time.sleep(2 ** attempt)
            continue

        error = response.json().get("error", {})
        raise RuntimeError(f"{error.get('code')}: {error.get('message')}")

    raise RuntimeError("Exhausted retries")

The rule of thumb: 402 is terminal, 429 means slow down by exactly the amount in Retry-After, and 5xx is retryable because failed requests are not charged.

Running queries in parallel

Keyword research means hundreds of queries, and the limit that matters is concurrency, not total volume. Bound it explicitly:

from concurrent.futures import ThreadPoolExecutor

keywords = ["serp api", "web scraping api", "proxy rotation", "headless browser"]

# Keep this at or below the concurrency allowed by your plan.
with ThreadPoolExecutor(max_workers=5) as pool:
    for keyword, data in zip(keywords, pool.map(search_safe, keywords)):
        print(keyword, len(data["organic_results"]))

Going above your plan’s concurrency does not make the job faster, it just converts requests into 429s.

What it costs

A search costs 5 credits, and every response tells you what it spent in the X-FastAF-Credits-Used header, so you can meter per feature rather than guessing after the fact. See pricing for the per-pack numbers, or the Google SERP API reference for the full parameter and response list.