Measure your brand's visibility in AI answers with Python

To measure a brand’s visibility in AI answers, run a fixed set of prompts through the AI surface several times, count the share of answers that name the brand with a word-boundary regex, and report an interval instead of a single number. A single run is not a measurement: on 19 October 2026 we sent 45 prompts through Google AI Mode three times each (135 answers), and for 43 of the 45 prompts the set of named brands differed between identical runs. The script below does the whole job in about 70 lines of Python, for 10 credits per answer.

Why is one run of an AI answer not a measurement?

Because the same prompt returns a different answer each time. Across the 135 answers, only 1 of 45 prompts produced three identical texts. Among the (brand, prompt) pairs where a brand appeared at least once, 138 of 302 (46%) were missing from at least one of the three runs. The answers are long (median 396 words) and name about five of the tracked brands each, so small changes in wording move brands in and out.

If you check your brand once a week and see it appear, then disappear, that is mostly this noise, not a change in your visibility. The AI Mode post covers the response format; this one is about what to do with many responses.

How was this measured?

Three product categories developers search for, 15 prompts each, phrased the way people type them: “best vector database”, “best vector database for rag”, “best async python web framework”, “web scraping api with a free tier”. No prompt names a brand. Each prompt went to /v1/ai with country=us and language=en, three times. 135 calls, 1,350 credits, 6.9 seconds on average (two confirmation calls to api.fastaf.co the same day took 6.1 and 3.6 seconds, each reporting X-FastAF-Credits-Used: 10). One call returned HTTP 200 with an empty answer and was charged 10 credits at the time; after this measurement we changed the rule, and an empty AI Mode answer is now refunded. Treated as data it would have counted as “no brand mentioned” for everyone, so the script still retries empty answers.

I picked 14 brands per category in advance. Mentions are counted in the answer text only (paragraphs, list items and table cells), not in cited page titles. “Position” is the order of first appearance among tracked brands. Nothing here is sentiment, an endorsement or a ranking of the products: it is what one Google surface said on one day.

What does the data show?

Mention rate is the share of the 45 answers naming the brand. The interval is a 95% bootstrap interval that resamples prompts, so it includes the effect of which 15 prompts you picked. Run 1 to 3 are the rates you would have reported from a single pass.

Category	Brand	Rate	95% interval	Run 1 / 2 / 3
Vector database	Qdrant	96%	87-100%	93 / 93 / 100
	Pinecone	84%	71-96%	80 / 80 / 93
	Weaviate	73%	56-91%	67 / 73 / 80
	Chroma	53%	31-73%	60 / 53 / 47
Python framework	Flask	91%	82-98%	80 / 93 / 100
	Starlette	49%	31-67%	53 / 40 / 53
	Litestar	38%	20-58%	40 / 27 / 47
Scraping API	Firecrawl	71%	51-91%	80 / 67 / 67
	ScraperAPI	47%	29-64%	53 / 40 / 47
	Apify	42%	24-60%	33 / 40 / 53

Three things stand out. First, brands in the middle of a category swing by 10 to 20 points between single passes (Apify 33% to 53%); the mean spread for brands above 20% was 11 to 14 points. Second, intervals from 15 prompts are plus or minus 20 points in the middle of the range, so a “+5 points” week-over-week change is invisible. Third, rankings by single pass differ: in the scraping category, ScraperAPI was third in pass 1, Apify replaced Oxylabs in the top five only in pass 3. Rate and position are also different things: Pinecone was named earlier (average position 2.2) than Qdrant (2.9) despite a lower rate.

Do the answers cite the brands’ own sites?

Mostly not. Of the distinct hostnames cited per answer, the share belonging to one of the tracked brands was 44% for scraping APIs, 30% for vector databases and 31% for Python frameworks. The rest were third parties: reddit.com appeared in 52 of the 135 answers (39%), then github.com, YouTube and comparison blogs. Who writes about you matters as much as what your own site says.

Sources also vary: two runs of the same prompt shared on average 28 to 34% of their cited sites (Jaccard overlap), so “which pages does the answer cite” needs repeats too.

How do I run this myself in Python?

Set FASTAF_API_KEY, fill PROMPTS with 15 or more prompts and BRANDS with the names you care about, then run it. Each call is appended to runs.jsonl, so a crash or a rerun skips what is already done.

import json, os, random, re, time
from concurrent.futures import ThreadPoolExecutor

import requests

API = "https://api.fastaf.co/v1"
HEADERS = {"Authorization": f"Bearer {os.environ['FASTAF_API_KEY']}"}
REPEATS = 3
PROMPTS = ["best vector database", "best vector database for rag", "best self hosted vector database"]  # use 15
BRANDS = {  # case-sensitive on purpose: "Falcon" is not "falcon"
    "Qdrant": r"\bQdrant\b", "pgvector": r"\bpgvector\b", "Milvus": r"\bMilvus\b",
    "Pinecone": r"\bPinecone\b", "Weaviate": r"\bWeaviate\b", "Chroma": r"\bChroma(?:DB)?\b",
}

def text_of(node):
    """Every visible string of the answer: paragraphs, list items, table cells."""
    if isinstance(node, str):
        return [node]
    if isinstance(node, list):
        return [s for v in node for s in text_of(v)]
    if isinstance(node, dict):
        return [s for k, v in node.items() if k in ("snippet", "text", "items", "table") for s in text_of(v)]
    return []

def ask(prompt):
    for attempt in range(5):
        r = requests.get(f"{API}/ai", headers=HEADERS, timeout=90,
                         params={"query": prompt, "country": "us", "language": "en"})
        if r.status_code == 200:
            text = "\n".join(text_of(r.json()["text_blocks"]))
            if text:  # an empty 200 is not "no brand mentioned": retry it
                return text
        time.sleep(3 * (attempt + 1))
    raise RuntimeError(f"gave up on {prompt!r}")

def collect(path="runs.jsonl"):
    done = {(r["prompt"], r["run"]) for r in map(json.loads, open(path))} if os.path.exists(path) else set()
    jobs = [(p, n) for p in PROMPTS for n in range(1, REPEATS + 1) if (p, n) not in done]
    def work(job):
        prompt, n = job
        row = {"prompt": prompt, "run": n, "text": ask(prompt)}
        with open(path, "a") as f:  # one line per finished call
            f.write(json.dumps(row) + "\n")
    with ThreadPoolExecutor(max_workers=5) as pool:
        list(pool.map(work, jobs))

def interval(per_prompt, n_boot=4000):
    """95% bootstrap interval resampling PROMPTS, keeping each prompt's runs together."""
    rng, k = random.Random(7), len(per_prompt)
    rates = sorted(sum(rng.choices(per_prompt, k=k)) / k for _ in range(n_boot))
    return rates[int(0.025 * n_boot)], rates[int(0.975 * n_boot)]

def report(path="runs.jsonl"):
    rows = [json.loads(line) for line in open(path)]
    print(f"{'brand':10}{'rate':>6}{'95% CI':>10}{'run1':>6}{'run2':>6}{'run3':>6}")
    table = []
    for brand, rx in BRANDS.items():
        hit = {(r["prompt"], r["run"]): bool(re.search(rx, r["text"])) for r in rows}
        per_prompt = [sum(hit[(p, n)] for n in range(1, REPEATS + 1)) / REPEATS for p in PROMPTS]
        by_run = [sum(hit[(p, n)] for p in PROMPTS) / len(PROMPTS) for n in range(1, REPEATS + 1)]
        table.append((sum(per_prompt) / len(PROMPTS), brand, interval(per_prompt), by_run))
    for rate, brand, (lo, hi), by_run in sorted(table, reverse=True):
        print(f"{brand:10}{rate:>6.0%}{f'{lo:.0%}-{hi:.0%}':>10}" + "".join(f"{x:>6.0%}" for x in by_run))

collect()
report()

With the three-prompt list above it printed, on 19 October 2026, pgvector, Weaviate, Qdrant and Milvus at 100% in every run, Chroma at 78% (67 / 67 / 100) and Pinecone at 67%.

Which counting mistakes change the numbers?

Case and word boundaries. Without \b, “Jina” matches inside longer words; without case sensitivity, “Bottle”, “Falcon” and “Reflex” match ordinary prose, and “scraper API” in lowercase counts as the product ScraperAPI. In my set, case-insensitive matching changed ScraperAPI from 21 to 22 answers and Zyte from 4 to 5, small here, but a lowercase “falcon” or “bottle” is easy to hit with a common-word brand. Add aliases you want (“ChromaDB”, “Bright Data” without the space) and keep them in a reviewed list. Also count only the answer, not the echoed query, or a prompt that contains your brand name measures your prompt.

For AI Overviews the same extraction works on the ai_overview object inside a /v1/serp response, which costs 5 credits instead of 10; that post covers presence and sources per keyword.

Does GEO change what to measure?

The GEO paper (Aggarwal et al., KDD 2024) reported up to 40% higher visibility in generative engine answers on its GEO-bench benchmark, but that is a tested set of content edits, not a guarantee for your site. Google’s guide to generative AI features, last updated 10 July 2026 as of this check, says “The best practices for SEO continue to be relevant” and that no special markup or llms.txt file is needed to appear in Google Search. So the work is still ordinary SEO; what you add is a way to tell whether it moved anything, and the answer above is: only with repeats and an interval.

What does it cost?

45 prompts x 3 runs is 1,350 credits, which is about $0.68 on the $10 Starter pack (20,000 credits). One 15-prompt category with 3 repeats is 450 credits. Failed requests are not charged. See pricing and the docs.

The short version

Count mentions with case-sensitive word-boundary regex, run every prompt at least three times, and report the mention rate with an interval that resamples prompts. In our 135 answers, 46% of brand-prompt appearances vanished in at least one repeat, and fewer than half the cited sources were the brands’ own. Start free with 500 credits and run the script on your own category.