How to Run an AI Agent Evaluation With the You.com Web Search API

TLDR: An AI agent evaluation scores how often your agent completes real tasks, not how well a single model answers a prompt. For agents that use the web, you evaluate two layers at once: the model's reasoning and the retrieval layer feeding it. The pattern that works is a fixed task set, a deterministic grader, pass@k across repeated trials, and a controlled variable. The You.com Web Search API plugs into that pattern as the retrieval arm you are testing, and the youdotcom-oss evals repositories give you runnable harnesses instead of a methodology you have to rebuild.

Agent evaluation is not model evaluation. A model eval asks whether one completion is correct. An agent eval runs a loop: the model plans, calls tools, reads results, and tries again, and any step can fail in a way the final answer hides. That is why agent evals score task completion over repeated trials, and why changing one variable at a time, the model, the prompt, or the search tool, is the only way to know what actually moved the number.

What Makes Agent Evaluation Different From Model Evaluation?

An agent evaluation must control three moving parts a model eval never sees: tool behavior, multi-step reasoning, and randomness.

Tool behavior means the same question can fail because retrieval returned nothing useful, even when the model reasons perfectly. Multi-step reasoning means a wrong turn at step two poisons step five, so you need end-to-end task scores, not per-step ones. Randomness means the same agent on the same task can pass once and fail once, so a single run tells you almost nothing.

The answer to randomness is pass@k: run each task k times and record the pass rate. The You.com agent evaluation harness reports passRate, passAtK, and passExpK per agent and tool combination (youdotcom-oss/web-search-agent-evals repository, fetched 2026-09-07), and its statistical analysis layer adds bootstrap confidence intervals, because a 70 percent pass rate on 20 tasks has wide enough error bars that ranking agents on it is guesswork.

What Should You Measure in an Agent Evaluation?

Four metrics carry almost all the signal: task completion, tool call efficiency, latency, and cost.

Task completion is the pass rate against a deterministic grader, the only metric that answers whether the agent works. Tool call efficiency is calls per completed task, which exposes agents that pass by brute force, burning ten searches to answer what one should. Latency is wall-clock time per task, which is what users actually feel. Cost is meter spend per task, which for search-backed agents is driven almost entirely by call volume.

The You.com evaluation guide adds a fifth lens for retrieval-heavy agents: measure the full workflow of search, synthesis, and grading together rather than the search API in isolation, because a mediocre synthesis prompt wastes a good retrieval layer (you.com/docs/guides/evaluate-us, fetched 2026-09-07).

How Do You Build an Agent Evaluation Harness?

The core loop is small: a task list, a runner, a grader, and a scorer. Here is a working pattern against the You.com Web Search API, adapted from the official evaluation guide's tool-calling example (you.com/docs/guides/evaluate-us, 2026-09-07).

import json, time
from youdotcom import You

def run_trial(task, k=5):
    passes = 0
    for _ in range(k):
        start = time.perf_counter()
        with You() as you:
            response = you.search(query=task["query"], count=10)
            snippets = [r.snippets[0] for r in response.results.web if r.snippets]
        answer = synthesize(task, snippets)  # your model + prompt
        grade = grade_answer(task["expected"], answer)  # deterministic judge
        passes += grade == "correct"
    return {"pass_rate": passes / k}

Three design choices matter more than the code. Keep the tool definition minimal, expose only the query parameter to the agent, because every extra parameter becomes a way for the model to misconfigure its own retrieval. Grade deterministically, with exact match, a regex, or a fixed rubric applied by code, because an LLM judge that varies run to run re-injects the randomness you designed the harness to remove. And log everything per trial: query, latency, result count, and grade, because the failure pattern lives in those logs, not in the aggregate score.

Which Datasets Should You Run Agent Evals On?

Start with public benchmarks that match your workload, then replace them with your own production tasks, which are the only set that predicts your outcomes.

The You.com evaluation guide recommends three public datasets by what they test (you.com/docs/guides/evaluate-us, 2026-09-07): OpenAI SimpleQA for fast factual questions as a baseline, FRAMES for multi-hop reasoning that mirrors agentic workflows, and FreshQA for time-sensitive queries that force the retrieval layer to earn its keep. The web-search-api-evals repository runs these plus DeepSearchQA, BrowseComp, and the FinSearchComp financial benchmarks, with providers integrated as swappable samplers (youdotcom-oss/web-search-api-evals repository, fetched 2026-09-07).

Your own task set should come from production logs, support tickets, or the questions your agent demonstrably fails at today. Fifty real tasks beat five hundred synthetic ones, and the evaluation guide is blunt about this: public benchmarks are the starting point, your production queries are the real test.

What Can You Copy Instead of Build?

You.com maintains open-source evaluation harnesses you can run as-is or fork, and each covers a different layer of the stack.

web-search-agent-evals is the agent layer: a matrix comparison that runs four coding agents (Claude Code, Gemini CLI, Droid, Codex) with builtin search versus the You.com MCP server in isolated Docker containers, 151 prompts with pass@k statistics and bootstrap confidence intervals (repository README, fetched 2026-09-07). If you are deciding whether to wire search into a coding agent, this harness answers it with data.

web-search-api-evals is the retrieval layer: it integrates You.com, Exa, Tavily, and Parallel as samplers, fetches results, synthesizes answers with an LLM, and grades against ground truth across SimpleQA, FRAMES, and the harder research benchmarks (repository README, fetched 2026-09-07). If you are choosing a search provider, this is the harness to run, and it is the honest way to compare vendors because you control the whole pipeline.

ydc-deep-research-evals is the synthesis layer: a pairwise comparison script that grades research reports on instruction following, comprehensiveness, completeness, and writing quality, with the DeepConsult dataset of business research queries (repository README, fetched 2026-09-07). Use it when your agent's output is a report rather than an answer.

What Failure Modes Ruin Agent Evals?

Four failure modes produce confident numbers that mean nothing.

Evaluating the search API alone. Retrieval quality in isolation does not predict task completion, because synthesis and prompting sit between them. Test the full chain, search then synthesis then grading, with your actual model and prompt.

Over-filtering the retrieval layer. Adding freshness, country, and safesearch filters to your first eval run adds three variables before you have a baseline. The evaluation guide's rule: run defaults first, add parameters only when the eval explicitly tests that feature.

Trusting a single run. One trial per task measures luck. Run k=5 minimum, and check the confidence interval before declaring a winner, especially on small task sets.

A drifting grader. An LLM judge prompted loosely will score the same answer differently across runs. Detection: grade a fixed set of answers twice and diff the scores. If they disagree, tighten the rubric or switch to deterministic grading.

How Does Retrieval Choice Change Agent Scores?

It changes them enough that the retrieval layer deserves its own arm in any agent eval. In the published results from the web-search-api-evals repository (fetched 2026-09-07), the same synthesis and grading pipeline produced materially different accuracy across search samplers on the same benchmarks, with FRAMES scores spanning from under 20 percent to over 70 percent depending on the sampler and configuration. The point is not which provider won on which day. The point is that the spread is large, which means retrieval choice is a first-order decision for agent quality, not a rounding error.

That is also why the You.com evaluation guide tells teams to compare within the same latency class: a sub-second search API and a multi-second deep research endpoint serve different agent loops, and cross-class comparisons mislead (you.com/docs/guides/evaluate-us, 2026-09-07).

Where Does This Fit With the Rest of the Tooling?

This article covered the evaluation layer. For the retrieval layer itself, the search API guide covers the Web Search API's parameters and response structure, and the LLM evaluation framework guide covers the model-side tooling that pairs with this harness. For methodological depth on why evals need statistical treatment, see Randomness in AI Benchmarks on the You.com resources hub.

Next action: clone the web-search-agent-evals repository, run its five-prompt smoke test against one agent with two search providers, and look at the pass@k spread. Then get an API key from the You.com platform and put your own production tasks into the prompt set, because that is the eval that actually predicts your outcomes.