Writing
Blog
Benchmarks we ran, things that broke, and how to scrape without babysitting proxies.
How an AI agent pays for a web search per call with x402
No account, no API key: the real 402 challenge FastAF returns, the Node client that pays it in USDC on Base, and how to cap what the agent can spend per call.
Amazon PA-API is retired: getting product data without it
PA-API 5 now answers 403 and points to the Creators API. Who should migrate, who needs a data API instead, a field mapping and real coverage numbers.
Amazon price comparison across EU countries, measured
We priced 80 products on amazon.de, .fr, .it, .es and .nl on 19 October 2026: median gap 19%, no single cheapest store. Method, tables and Python.
Build an MCP server in Python that wraps a web data API
A working MCP server in 109 lines of Python: three web data tools, errors that tell the agent what to do, and a per-session credit budget. Tested on 19 Oct 2026.
Build a web research agent with the Claude Agent SDK and a hard budget
About 60 lines of Python: Claude Agent SDK plus the FastAF MCP server for search, page reads and Google Scholar. One real run cost 11 credits.
Is web scraping legal in the EU? GDPR, database rights, AI Act
What EU law says about scraping public web pages: GDPR and legitimate interest, the database right, text and data mining opt-outs, the AI Act, and a checklist.
LangChain’s Google search tool after Custom Search shuts down
Replace GoogleSearchAPIWrapper before Google Custom Search ends on 1 January 2027: a tested LangChain tool that keeps title, link and snippet.
Built-in LLM web search vs a SERP API: cost per answer
Claude and OpenAI charge $10 per 1,000 searches plus the result tokens. We measured the token side on 20 questions and priced 1,000 answers both ways.
Add web search to Claude Code, Cursor and Codex with one MCP server
One remote MCP server gives a coding agent 15 tools: web search, page scraping to markdown, maps, Amazon, YouTube transcripts. Setup for three clients, tested.
n8n web scraping and Google search: a workflow without code
Import one n8n workflow that loops over keywords, calls a Google search API, scrapes the top pages to markdown and writes rows to Google Sheets or a CSV.
Playwright MCP token cost vs reading pages as markdown
We drove the real Playwright MCP server and FastAF markdown over 11 public pages and counted tokens. Median page: 9,874 vs 5,083, with exceptions both ways.
Measure your brand's visibility in AI answers with Python
We ran 45 prompts through Google AI Mode three times each. 43 of 45 changed which brands they named between identical runs. Method, numbers and a Python script.
Track Google AI Overviews in Python: presence, sources, changes
We checked 100 Google queries for AI Overviews on 19 October 2026: who gets one, what it cites, how stable it is, plus a Python tracker that stores snapshots.
Web search for AI agents after Bing: what options are left in 2026?
Bing Search APIs retired in August 2025 and Google Custom Search is closed to new customers. Four kinds of search backend remain for agents.
Why your AI agent’s fetch tool comes back with an empty page
Empty shell, cookie wall, login redirect or a page cut short: how to tell them apart in 30 lines of Python, with real measurements from 19 October 2026.
Getting Google's AI Mode answer and its sources over an API
The composed answer and the pages it cites, as JSON, and why the wording is Google Search AI Mode output that can change between two identical calls. Tested August 2026.
Pulling papers, authors and citation counts from Google Scholar with Python
Query Google Scholar for a topic and get papers, years and citation counts as JSON, built around an endpoint slow enough that it has to run as a batch job. Tested August 2026.
Compare product prices across merchants with the Google Shopping API
Parse the price string yourself, discard the parts of the payload you do not need, and query narrow enough to get like-for-like offers. Tested in August 2026.
Turn Google Trends into a demand signal with Python
Read the time series, compute a simple trend direction, and the one thing to get right: a Trends value is relative interest, never a search-volume number. Tested August 2026.
How to scrape Google Jobs listings with Python
Company, location and apply links from Google Jobs as JSON, and the one response shape that breaks naive parsing code on a plain HTTP 200. Tested August 2026.
Track brand and competitor mentions with the Google News API
Poll a set of query terms on a schedule, dedupe by article link, and see only what changed since the last run. Tested against a real response in August 2026.
Monitor Walmart prices for a set of products with Python
Build a Walmart search URL, walk the nested groups and items in the response, and pair it with the Amazon endpoint to compare a price across both. Tested August 2026.
Automate website screenshots with Python for monitoring
Capture a page as a PNG with one request, write the bytes straight to a file, and branch on content-type instead of guessing when a capture fails. Tested August 2026.
Get public X profile data with Python for lead enrichment
Look up follower counts, bio and verification for a public X profile with one request and no developer account, plus exactly what the endpoint does not cover. Tested August 2026.
Concurrency, not credits, decides how long your scraping job takes
The same 200,000-request job takes 22 hours at 5 concurrent slots, 4.4 hours at 25. Why plans cap concurrency, and a correct worker pool in Python.
How much does it cost to scrape 1 million pages in 2026?
DIY proxies, compute and engineer time versus a flat $299 for 1,000,000 credits: the real cost to scrape 1 million pages, checked August 2026.
Free SERP API tiers compared (2026)
SerpApi gives 250 searches a month, Serper gives 2,500 once, FastAF gives 500 credits (100 searches) once. Checked on each pricing page, August 2026.
Google Maps lead generation: a practical playbook
A category and a city list, turned into a filtered, enriched CSV of leads. Full Python loop and the credit cost of a 100-city campaign, August 2026.
How to get Google search results as JSON (no HTML parsing)
Google serves obfuscated, rotating markup. Here is the /serp JSON contract instead: guaranteed fields, optional blocks, tested August 2026.
Google's official Search API vs SERP APIs: what are your options in 2026?
The Custom Search JSON API gives 100 free queries a day and $5 per 1,000, capped at 10k a day, and is closed to new signups as of August 2026.
How to monitor keyword rankings with Python
A minimal rank tracker in Python: query, pull your position from organic_results, store it in SQLite, and the credit math behind tracking depth.
Is LinkedIn scraping legal in 2026?
hiQ v. LinkedIn set the CFAA baseline for public data, but LinkedIn settled a new scraping lawsuit in February 2026. What that means for you, in plain terms.
JavaScript rendering in web scraping: when you need it and what it costs
How to tell if a page needs a real browser instead of a plain fetch, why rendering costs more everywhere, and the credit multipliers, August 2026.
Build a price tracking bot in Python (Amazon + Google Shopping)
Discover products, track ASINs by price, store history in SQLite, and alert on drops. A full price bot with the real credit cost per product, August 2026.
requests + BeautifulSoup vs a scraping API: when DIY is enough
For static sites with no anti-bot, requests and BeautifulSoup cost zero and work fine. Here is exactly where that stops being true.
How to scrape Amazon product data without getting blocked
Search results, prices, ratings and ASINs from Amazon as JSON: why the DIY approach fails fast, what the data costs, and a tested Python example. August 2026.
How to scrape any website with Python in 2026
A general /scrape workflow: static fetch by default, escalate to JS rendering and premium proxies only when needed, then parse with BeautifulSoup.
How to scrape Google Maps results with Python
Get places, ratings, review counts and contact details from Google Maps as JSON, why the official Places API gets expensive, and what a Maps search actually costs. With a tested example, August 2026.
Scraping Google Search without getting blocked: what actually happens
Rate limits, CAPTCHA walls, fingerprinting and pagination math: the actual mechanics behind why raw Google scraping breaks, and why proxies alone do not fix it.
How to get YouTube transcripts with Python (without API quotas)
The YouTube Data API skips transcripts, and youtube-transcript-api gets IP blocked on cloud servers. A setup that costs 1 credit per video, August 2026.
Giving an LLM fresh web data: SERP APIs for RAG and grounding
A minimal Python retrieval pipeline for grounding an LLM in live search results, and why a fixed credit cost beats an unpredictable per-query bill.
Give an AI agent live web search with a hard spending cap
An agent that can search the web can also loop. Here is how to wire live search into an agent so the budget is enforced by the API, not by hoping the model behaves.
What a SERP API actually costs per 1,000 searches
Credits, plan tiers and pagination make search API pricing hard to compare. Here is how to work out your real cost per 1,000 searches, with published list prices as of August 2026.
How to scrape Google search results in Python
Why requests + BeautifulSoup stops working on Google, and a working Python setup that returns parsed SERP JSON with pagination, country targeting and sane error handling.