Concurrency, not credits, decides how long your scraping job takes

Two accounts can have the exact same credit balance and finish the same job at wildly different speeds. The number that decides how long a batch job runs is concurrency, how many requests you’re allowed to have in flight at once, and it’s usually not the number people check first.

The math

Say a job is 200,000 requests, and each one takes about 2 seconds round trip, a reasonable average for a rendered page or a search with retries. Total work is 400,000 seconds. How long that takes to run depends entirely on how many of those seconds you can spend in parallel:

Concurrent slots	Total time
1	111.1 hours (4.6 days)
5	22.2 hours
10	11.1 hours
25	4.4 hours

Same job, same credit cost, a 25x difference in wall-clock time between 1 slot and 25. If a nightly job needs to finish before a business day starts, concurrency is the number to check before you commit to a schedule, not the credit balance.

Why plans cap concurrency, not just credits

Credits meter how much work you’re allowed to do in total. Concurrency meters how much of that work can happen at the same instant, and it exists for a separate reason: every request in flight holds a browser session, a proxy connection or both, on infrastructure that costs money to keep provisioned regardless of whether you’re near your credit limit.

On FastAF, concurrency scales with the credit pack you buy: the free tier runs 2 concurrent requests, every paid pack 5 (Starter, Growth, Scale and Pro), Business 25 (October 2026). A Free-tier account can run two requests at once; a Business account can have 25 open simultaneously. Sending more requests than your pack’s concurrency doesn’t queue politely and go faster, it starts returning 429 responses, so the cap is worth designing your job around rather than discovering by trial and error.

Writing a worker pool that respects the cap

The mistake that actually costs you time is setting max_workers optimistically and letting the retry logic paper over the resulting 429s. Set it to your pack’s actual concurrency and the 429s mostly stop happening in the first place.

ThreadPoolExecutor, the simpler option for most scripts:

import time
import requests
from concurrent.futures import ThreadPoolExecutor

API_KEY = "faf_live_..."  # your FastAF key
CONCURRENCY = 5  # match your pack: Free 2, Starter to Pro 5, Business 25

def fetch(url, max_retries=5):
    for attempt in range(max_retries):
        resp = requests.get(
            "https://api.fastaf.co/v1/scrape",
            params={"url": url},
            headers={"Authorization": f"Bearer {API_KEY}"},
            timeout=90,
        )
        if resp.status_code == 200:
            return resp.json()
        if resp.status_code == 429:
            wait = int(resp.headers.get("Retry-After", 2 ** attempt))
            time.sleep(wait)
            continue
        if resp.status_code in (502, 503, 504):
            time.sleep(2 ** attempt)  # transient upstream failure, not billed
            continue
        resp.raise_for_status()
    raise RuntimeError(f"gave up on {url} after {max_retries} attempts")

urls = [...]  # your 200,000 URLs
results = []
with ThreadPoolExecutor(max_workers=CONCURRENCY) as pool:
    for result in pool.map(fetch, urls):
        results.append(result)

asyncio, if the rest of the pipeline is already async, with a semaphore doing the same job max_workers did above:

import asyncio
import httpx

CONCURRENCY = 5
sem = asyncio.Semaphore(CONCURRENCY)

async def fetch(client, url, max_retries=5):
    async with sem:
        for attempt in range(max_retries):
            resp = await client.get(
                "https://api.fastaf.co/v1/scrape",
                params={"url": url},
                headers={"Authorization": f"Bearer {API_KEY}"},
                timeout=90,
            )
            if resp.status_code == 200:
                return resp.json()
            if resp.status_code == 429:
                wait = int(resp.headers.get("Retry-After", 2 ** attempt))
                await asyncio.sleep(wait)
                continue
            if resp.status_code in (502, 503, 504):
                await asyncio.sleep(2 ** attempt)
                continue
            resp.raise_for_status()
        raise RuntimeError(f"gave up on {url} after {max_retries} attempts")

async def run(urls):
    async with httpx.AsyncClient() as client:
        return await asyncio.gather(*(fetch(client, u) for u in urls))

results = asyncio.run(run(urls))

The semaphore holds the exact same concurrency ceiling as max_workers above; pick whichever style matches the rest of your codebase, not one for correctness reasons.

The two things that actually cause 429s in practice

Workers above the pack’s cap. The obvious one: max_workers=50 on a Starter pack (5 concurrent) means 45 of every 50 requests immediately fail with 429, and the backoff logic just makes that slow instead of fast.

A second process you forgot about. A cron job and a manual test run against the same API key at the same time double your effective concurrency without either process knowing about the other. If you’re seeing 429s despite max_workers being set correctly, check for a second script running against the same key before you assume the cap changed.

The short version

Before estimating how long a batch job will take, check your pack’s concurrency, not just its credit balance: 2 for Free, 5 for Starter, Growth, Scale and Pro, 25 for Business as of October 2026. Set your worker pool’s size to that number, honor Retry-After on 429, and back off with exponential delay on 5xx. A job’s runtime is (total requests x average latency) / concurrent slots, and concurrency is the only one of those three you can actually change by buying a bigger pack.