Web Search API Evaluation: How to Benchmark a Search Provider Before You Commit

TLDR: A web search API evaluation should answer a purchasing question: which provider meets your application's accuracy, latency, reliability, and cost requirements? Freeze a representative query set, compare equivalent search configurations, preserve the evidence supplied to synthesis, and count failed requests. Use paired comparisons with uncertainty rather than treating a leaderboard or a small pilot as a procurement decision.

This guide focuses on selecting a search provider, not choosing an evaluation framework or scoring an entire autonomous agent. The unit of comparison is a specified search configuration inside your application pipeline. A useful result identifies both the winning configuration and the conditions under which that conclusion holds.

What exactly should the benchmark measure?

Write a decision contract before collecting results. State the target workload, maximum acceptable response time, minimum answer quality, expected traffic, and cost ceiling. Define disqualifying failures, such as unsupported answers to high-risk questions. Set thresholds from product requirements rather than choosing thresholds after seeing which vendor wins.

Measure search usefulness through the answer your application produces, while retaining retrieval diagnostics. An irrelevant source list and an incorrect answer require different fixes. The official You.com evaluation guide recommends testing the full search, synthesis, and grading workflow with your actual model and prompts, starting with simple settings and adding parameters for explicitly tested needs.

Run two clearly labeled comparisons if necessary. A controlled comparison equalizes downstream processing and evidence budgets. A deployment comparison tests each provider's intended production configuration within the same service and spending limits. The latter may inform purchasing more directly, but it cannot isolate retrieval quality from configuration differences.

Freeze a dataset that represents the purchase

Build a development set for tuning and a separate held-out decision set. Sample permitted production queries across meaningful slices: factual lookup, recent events, difficult entity disambiguation, region, language, and questions requiring multiple sources. Remove personal information and near-duplicates. Keep related questions together when separating development and test data to reduce leakage.

Give every question a stable identifier, reference answer, supporting evidence, and an answer-validity timestamp. Record sampling rules, slice counts, dataset version, and a file checksum. Keep evaluation questions out of synthesis prompts except as the question itself; never expose reference answers to the answering model. Time-sensitive references need explicit cutoff dates and independent review.

Use public datasets as additional diagnostics, not substitutes for workload coverage. The official guide recommends SimpleQA for factual questions, FRAMES for multi-step reasoning, and FreshQA for time-sensitive queries. However, FreshQA is absent from the repository's current dataset registry. Using it there requires additional dataset loading and grading integration, not merely selecting an existing flag.

A small pilot is useful for finding broken adapters and estimating disagreement rates. It is not conclusive selection evidence. Choose the decision-set size around the improvement that would change your purchase, the frequency of critical slices, and the precision you need. Reserve additional data rather than repeatedly tuning against the same final test.

Make provider configurations comparable

Freeze query text, synthesis model version, prompt, generation settings, grader, rubric, and response format. Declare result count, accepted content types, extraction mode, and the maximum evidence budget passed downstream. Preserve source URLs, titles, snippets, and extracted content where available. Log truncation and ordering rules instead of silently reducing rich results to titles and descriptions.

Equivalent budgets do not require pretending endpoints are interchangeable. Document unsupported filters or extraction features. First compare the common configuration; then run separately labeled feature tests that reflect actual deployment. The official guide's defaults-first recommendation is a useful baseline discipline, not a reason to ignore a required region or freshness constraint.

Run providers against the same query identifiers within a bounded time window. Randomize or alternate provider order, equalize concurrency, and record client region and connection reuse. Save request parameters, response timestamps, raw evidence, final answers, grades, and model versions in an access-controlled run directory. This makes failures explainable without exposing sensitive logs in a public report.

Holding synthesis fixed improves attribution, but it does not prove every score difference comes from retrieval. Search results interact with prompts and context limits; network conditions, index changes, and model randomness remain possible influences. Repeat paired trials and inspect disagreements before calling a difference a durable provider advantage.

What do the published repository results actually establish?

The vendor-published repository results report SimpleQA accuracy from 59.11% to 92.09% across the listed configurations. Its FRAMES search-only entries range from 19.30% to 39.93%. The separate 70.75% FRAMES entry is You.com Research Lite, not an equivalent search-only configuration. These are published measurements, not results independently reproduced for this article.

That distinction is implemented in source. The sampler registry assigns Research Lite to the research sampler. The research sampler defaults to no shared synthesis, and the processing pipeline uses its formatted answer directly. Do not describe that research score as retrieval alone under identical shared synthesis.

Two reporting details also matter. The README uses provider-reported internal latency when available and request duration otherwise. Its results analyzer excludes rows marked FAILED in the answer or evaluation from the accuracy denominator. Consequently, published accuracy is not automatically the failure-inclusive service metric your procurement decision needs.

Track a compact, auditable scorecard

Metric	Recommended definition	Decision value
Failure-inclusive accuracy	Correct answers divided by all scheduled queries	Counts inability to deliver an answer
Graded-only accuracy	Correct answers divided by successfully graded queries	Separates answer quality from execution failures
Failure rate	Terminal failures divided by scheduled queries, split by cause	Exposes timeouts, errors, and grading gaps
Client latency	Median and p95 elapsed duration under a declared timeout policy	Shows typical and tail experience
Cost	Total measured spend per scheduled query and per correct answer	Includes unsuccessful work

Keep search latency separate from end-to-end answer latency. Use a monotonic client clock consistently, including network time, retries, and backoff within the measured operation. Log timeout duration even when no answer arrives. A timeout is a terminal observation at your policy limit, not evidence that the provider would have completed at that time.

Count search, extraction, synthesis, and grading spend separately, then report the relevant deployment total. Evaluation-only grading costs should not be disguised as production serving costs. Record the pricing date, units, discounts, and whether figures are measured or estimated. Unknown charges are missing data, not zero. If no answer is correct, cost per correct answer is undefined.

For retrieval diagnosis, review whether returned evidence actually supports the reference answer, whether sources meet the task's authority requirements, and whether citations support the generated claims. Define those rubrics before review. An accurate answer can still cite an irrelevant page; a relevant page can still be misread during synthesis.

Declare retries, caching, and failure accounting

Neither retries nor caching is inherently unfair. Specify retryable conditions, maximum attempts, backoff, and an overall deadline for every provider. Log each attempt's status and cost, then aggregate the full user-visible duration. Equal policy does not guarantee equal retry counts; that difference is part of the operational result.

Separate cold-cache and production-cache experiments. Document cache keys, retention, prewarming, and hit rates. Do not warm one provider with evaluation queries while another starts cold. Provider-side caching may be unobservable, so distinguish what you controlled from what you could only record. Repeated runs should keep the declared policy unchanged.

Maintain a manifest of every scheduled provider-query pair. Never drop a timeout or parsing failure because no answer exists. Record grading failures separately from provider failures so an unavailable judge does not masquerade as poor retrieval. Use a conservative failure-inclusive score while grading remains unresolved, then report any regrading as a versioned correction.

Turn the protocol into a reproducible run

1. Lock the inputs: export the held-out manifest, freeze its checksum, and save the exact provider settings, prompts, and model identifiers before starting collection.
2. Check adapters: use development questions to verify authentication, evidence serialization, timing boundaries, and failure logging. Confirm that every adapter preserves the source material available to synthesis.
3. Collect paired trials: run the scheduled provider-query pairs with the declared ordering and concurrency policy. Write terminal records even after exhausted retries, and retain attempt-level evidence separately.
4. Grade and reconcile: apply the frozen rubric, audit disagreements, and match the logs against the manifest. Investigate missing rows rather than silently filling them with invented latency or costs.
5. Report and decide: calculate the predefined metrics, uncertainty, and slice results. Archive the raw run before making configuration changes, then use a new run identifier for any follow-up experiment.

Predefine how an answer earns credit. A factual lookup might require an acceptable answer alias and a supporting source; a multi-source question might require every requested component. Decide whether partial credit is allowed before reviewing outputs. When the final metric is binary, document how partial answers map to that decision rather than letting graders improvise provider-specific standards.

Report both workload-weighted performance and important slice results. Oversampling rare, difficult questions helps diagnosis, but its unweighted average may no longer represent production traffic. Preserve the sampling weights and show slice sizes. A strong overall mean should not conceal an unacceptable failure pattern in the language or jurisdiction that matters to the buyer.

Run this offline scoring check before spending on APIs

The following example validates your accounting, not provider performance. It needs Python 3 and no external packages, credentials, or network calls. Save the three blocks as dataset.jsonl, logs.jsonl, and score_search_logs.py in one directory. All questions, grades, prices, and durations below are synthetic and illustrative; they are not benchmark claims.

The dataset is the fixed manifest. Logs contain exactly one terminal row per provider and query for one paired trial. Status is ok, error, timeout, or grade_error; correct must be a boolean for ok and null otherwise. Latency is total client-observed operation time in milliseconds, and cost is total recorded dollars for that operation, including retries. Correctness is supplied by an upstream rubric or reviewer, not invented by this aggregator.

{"query_id":"q1","query":"What is two plus two?","expected":"4"}
{"query_id":"q2","query":"What is three plus three?","expected":"6"}
{"query_id":"q3","query":"What is four plus four?","expected":"8"}

{"provider":"A","query_id":"q1","status":"ok","correct":true,"latency_ms":100,"cost_usd":0.01}
{"provider":"A","query_id":"q2","status":"ok","correct":true,"latency_ms":200,"cost_usd":0.01}
{"provider":"A","query_id":"q3","status":"timeout","correct":null,"latency_ms":1000,"cost_usd":0.01}
{"provider":"B","query_id":"q1","status":"ok","correct":true,"latency_ms":120,"cost_usd":0.02}
{"provider":"B","query_id":"q2","status":"ok","correct":false,"latency_ms":160,"cost_usd":0.02}
{"provider":"B","query_id":"q3","status":"ok","correct":true,"latency_ms":300,"cost_usd":0.02}

"""Offline aggregation only. Python 3 standard library; no API calls."""
import json
import math
import random
import statistics
import sys
from pathlib import Path


def load(path):
    return [json.loads(line) for line in Path(path).read_text().splitlines()
            if line.strip()]


def require(condition, message):
    if not condition:
        raise ValueError(message)


def percentile(values, probability):
    ordered = sorted(values)
    return ordered[max(0, math.ceil(probability * len(ordered)) - 1)]


def summarize(dataset_path, logs_path, provider_a, provider_b):
    require(provider_a != provider_b, "Choose two distinct providers")
    dataset = load(dataset_path)
    ids = [row["query_id"] for row in dataset]
    require(ids and all(isinstance(q, str) and q for q in ids), "Invalid query IDs")
    require(len(ids) == len(set(ids)), "Duplicate dataset IDs")
    for row in dataset:
        require(all(isinstance(row.get(k), str) and row[k]
                    for k in ("query", "expected")), "Invalid dataset row")
    providers = [provider_a, provider_b]
    index = {}
    for row in load(logs_path):
        key = (row["provider"], row["query_id"])
        require(key[0] in providers and key[1] in ids, "Unexpected log key")
        require(key not in index, "Duplicate log row")
        require(row["status"] in ("ok", "error", "timeout", "grade_error"),
                "Invalid status")
        if row["status"] == "ok":
            require(type(row["correct"]) is bool, "OK rows need boolean grades")
        else:
            require(row["correct"] is None, "Failed rows need null grades")
        for field in ("latency_ms", "cost_usd"):
            value = row[field]
            require(type(value) in (int, float) and math.isfinite(value)
                    and value >= 0, "Invalid " + field)
        index[key] = row
    require(set(index) == {(p, q) for p in providers for q in ids},
            "Missing scheduled query: log its terminal failure, do not drop it")
    scores, output = {}, {}
    for provider in providers:
        rows = [index[(provider, q)] for q in ids]
        scores[provider] = [int(r["status"] == "ok" and r["correct"])
                            for r in rows]
        correct = sum(scores[provider])
        graded = sum(r["status"] == "ok" for r in rows)
        cost = sum(r["cost_usd"] for r in rows)
        output[provider] = {
            "scheduled": len(ids), "graded": graded,
            "failed": len(ids) - graded, "correct": correct,
            "failure_inclusive_accuracy": correct / len(ids),
            "graded_only_accuracy": correct / graded if graded else None,
            "failure_rate": (len(ids) - graded) / len(ids),
            "latency_p50_ms": percentile([r["latency_ms"] for r in rows], 0.50),
            "latency_p95_ms": percentile([r["latency_ms"] for r in rows], 0.95),
            "mean_cost_usd": cost / len(ids),
            "cost_per_correct_usd": cost / correct if correct else None,
        }
    differences = [a - b for a, b in zip(scores[provider_a], scores[provider_b])]
    rng = random.Random(2026)
    draws = [statistics.mean(rng.choices(differences, k=len(ids)))
             for _ in range(2000)]
    output["paired_A_minus_B"] = {
        "A": provider_a, "B": provider_b,
        "accuracy_difference": statistics.mean(differences),
        "percentile_bootstrap_95_interval": [percentile(draws, 0.025),
                                              percentile(draws, 0.975)],
        "resampling_unit": "query", "draws": 2000, "seed": 2026,
    }
    return output


if __name__ == "__main__":
    if len(sys.argv) != 5:
        raise SystemExit("Usage: python3 score_search_logs.py dataset.jsonl logs.jsonl A B")
    print(json.dumps(summarize(*sys.argv[1:]), indent=2, allow_nan=False))

Run the local scorer with this command:

python3 score_search_logs.py dataset.jsonl logs.jsonl A B

The scorer rejects duplicate, missing, and unexpected provider-query pairs, invalid grades, and nonfinite or negative measurements. It reports failures, two accuracy denominators, nearest-rank latency percentiles, costs, and a paired bootstrap interval. With the illustrative input, A has two correct answers and one timeout; B has two correct answers and one incorrect answer. Both have the same failure-inclusive accuracy, but their reliability and conditional accuracy differ.

Quantify uncertainty before choosing a supplier

Compare outcomes query by query. The script resamples paired query differences together, preserving the pairing between providers, and reports a percentile bootstrap interval. Its tiny fixture demonstrates mechanics only. Three synthetic questions cannot justify a purchasing conclusion, and the interval does not capture index drift or variation across collection days.

For repeated trials, keep every provider's outcomes paired within the same query and trial. Aggregate within queries or use a clustered bootstrap that resamples whole queries with their repeated observations. If questions share an entity or source, consider that larger dependency cluster. Treating repeated answers as independent questions can make uncertainty look smaller than it is.

Blind human reviewers to provider identity when practical, audit a fixed sample of grader decisions, and examine disagreement cases. Regrading stored answers measures judge instability separately from recollecting search results. Read Randomness in AI Benchmarks for related methodology, and How We Evaluate AI Search for the broader search evaluation approach.

Finally, apply the thresholds you wrote first. Prefer a configuration that clears essential quality and reliability gates within budget over a marginal average winner with unresolved critical failures. Archive the dataset checksum, configuration, evidence, scoring output, uncertainty, and known limitations with the decision. Expand an inconclusive test rather than manufacturing certainty from a narrow lead.