Web Search API in Python: A Practical Guide With the You.com SDK

TLDR: Use the official youdotcom Python SDK to search the web, inspect typed results, and retrieve page content. This guide targets SDK 3.3.0 and Python 3.10 or later. Every example supplies an API key from the environment, sets an explicit timeout, and keeps requests inside an open client context. Start with ordinary search, add extraction when you need more text, and configure retries deliberately rather than assuming they happen automatically.

A web search API connects a Python application to information beyond its local files or model training data. The important integration work is not just sending a query. It is preserving source URLs, choosing the right amount of evidence, recognizing missing content, and deciding how failures affect the application. Search and Contents solve different steps: Search discovers URLs from a query; Contents reads URLs you already have.

How do you install and authenticate the Python SDK?

The 3.3.0 release on PyPI requires Python 3.10 or later. Install it with python -m pip install youdotcom==3.3.0 in your project's virtual environment. Pinning the version makes these examples a reproducible starting point; review release changes and rerun your tests before upgrading. The official SDK README documents authentication through an API key sent in the X-API-Key header.

Set YDC_API_KEY through your shell or deployment secret manager before running a snippet. Do not paste a real key into source code, a notebook, or an issue report. These examples intentionally fail before opening a client when the variable is absent or blank. That makes a configuration problem visible locally instead of confusing it with a remote search failure.

The SDK also supports automatic lookup: omitting api_key_auth or passing None reads YDC_API_KEY, then the legacy YOU_API_KEY_AUTH variable. A nonempty explicit key takes precedence. An explicitly empty or blank value raises ValueError rather than silently selecting another identity. The explicit check below chooses a stricter application policy: require YDC_API_KEY and do not use the legacy fallback.

What does a first search call look like?

This standalone example requests five results per section and prints web sources with their available snippets. It does not add domain, country, or freshness filters, so it provides a baseline before you narrow the query. Its ten-second timeout is an example client configuration, not a measured endpoint latency or a service guarantee.

import os
from youdotcom import You

key = os.environ.get("YDC_API_KEY")
if not key or not key.strip():
    raise RuntimeError("Set a nonblank YDC_API_KEY before running")

with You(api_key_auth=key, timeout_ms=10_000) as you:
    response = you.search(query="Python asyncio task cancellation", count=5)
    web = (response.results.web or []) if response.results else []
    for hit in web:
        print(hit.title or "Untitled", hit.url)
        for snippet in hit.snippets or []:
            print(snippet)

Keep all calls inside the context manager. The SDK lifecycle documentation says the client is not reusable after exit. A later Contents call therefore needs either the same still-open block or its own client. For a service, design an explicit client lifetime instead of copying a variable from a notebook cell whose context has already closed.

Which response fields should your application keep?

The SearchResponse model makes results and metadata optional; the nested web and news lists are optional too. Guard both results and web before iterating. An empty list should produce an explicit no-results state, not an invented answer or a crash. Keep missing results separate from request exceptions in your metrics so successful empty responses do not hide authentication failures.

Web results expose fields including URL, title, description, snippets, page age, thumbnail URL, favicon URL, and optional extracted contents. Do not require every presentation field to be populated. Preserve the URL alongside the exact evidence you pass downstream. Flattening everything into a title and description discards passages that may support the final answer, and makes later citation review harder.

Text level	Field to read	Choose it when
Default snippets	hit.snippets	You need short fragments for browsing results.
Highlights	hit.contents.highlights	You need query-relevant passages for grounding.
Full page	hit.contents.markdown or hit.contents.html	You need the document rather than selected passages.

The Search guide distinguishes these text levels. Highlights replace snippets rather than supplementing them. Full-page mode can return snippets plus extracted page content. Build your evidence adapter around the requested mode and guard contents before reading any nested field. For a production pipeline, retain the mode with each stored result so missing snippets are not mistaken for broken parsing.

How do filters and nested extraction work?

Search accepts freshness, country, language, safe-search, and domain controls. For example, freshness="week" selects the last seven days, while country="US" requests geographic targeting. Apply filters only when they express a real requirement. A recent-results filter is useful for an update feed but can exclude the foundational documentation a troubleshooting question needs. Compare filtered results against your unfiltered baseline before making a filter universal.

The README documents an important domain constraint: include_domains cannot be combined with exclude_domains or boost_domains; the API returns 422 for that combination. An allowlist restricts eligible sources, whereas boosting changes preference without excluding everything else. Keep these options in your application configuration with tests for mutually incompatible combinations rather than scattering them across call sites.

Extraction is a nested object, not a mode string at the top level. In the following standalone example, extraction_mode and extraction_source belong inside extraction, while extraction_formats belongs inside extraction.full_page. crawl_timeout remains a search argument measured in seconds. The extraction model defines this shape and rejects unknown nested keys.

import os
from youdotcom import You

key = os.environ.get("YDC_API_KEY")
if not key or not key.strip():
    raise RuntimeError("Set a nonblank YDC_API_KEY before running")

with You(api_key_auth=key, timeout_ms=30_000) as you:
    response = you.search(
        query="Python asyncio task cancellation",
        count=3,
        include_domains=["docs.python.org"],
        extraction={
            "extraction_mode": "full_page",
            "extraction_source": "blend",
            "full_page": {"extraction_formats": ["markdown"]},
        },
        crawl_timeout=10,
    )
    web = (response.results.web or []) if response.results else []
    for hit in web:
        text = hit.contents.markdown if hit.contents else None
        if not text:
            print("No extracted Markdown:", hit.url)
            continue
        print(hit.url, text)

For full-page extraction, blend serves cached content when available and fetches live on a miss. cache uses cached content only; a cache miss can leave contents absent. fetch always crawls live. Choose the policy based on freshness requirements and measure its effect instead of assuming every result will contain a page. A missing body does not invalidate the URL or justify silently treating a snippet as the full document.

To request highlights instead, replace the entire extraction dictionary with {"extraction_mode": "highlights"}. Read contents.highlights only after checking contents, and treat its missing list as empty. Do not carry extraction_source or full_page into that mode. The SDK strips crawl_timeout from highlights requests, and the current extraction object replaces the older livecrawl parameters; do not combine both interfaces.

When should you use the Contents API?

Use Contents when you already know the pages, or when you want to inspect only selected search hits. The Contents guide documents batches of up to ten URLs, Markdown or HTML output, and per-URL crawl timeouts. A two-stage workflow lets your application decide which discovered sources deserve a full read instead of extracting every result automatically.

import os
from youdotcom import You

key = os.environ.get("YDC_API_KEY")
if not key or not key.strip():
    raise RuntimeError("Set a nonblank YDC_API_KEY before running")

with You(api_key_auth=key, timeout_ms=30_000) as you:
    pages = you.contents(
        urls=["https://docs.python.org/3/library/asyncio-task.html"],
        formats=["markdown"],
        crawl_timeout=10,
        max_age=86_400,
    )
    for page in pages or []:
        if not page.markdown:
            print("No readable Markdown:", page.url)
            continue
        print(page.title or "Untitled", page.url)
        print(page.markdown)

max_age is a cache-age threshold in seconds, not the client's HTTP timeout. Here it rejects cached content older than one day; zero forces a fresh fetch, while leaving it unset accepts cached content regardless of age. A blocked or failed page can have null Markdown or HTML, so process each page independently and record missing content instead of discarding the entire successful batch.

This example requests only Markdown. The current README marks the metadata format deprecated, although the released SDK enum and Contents guide still list it. It has not already disappeared from the release. Its documented fields are site_name and favicon_url, not a general structured-data payload. Avoid building a new dependency on that deprecated format when the task is simply reading page text.

How do you handle search errors and opt into retries?

Classify failures before choosing a retry policy. Search exposes UnauthorizedResponseError for 401, ForbiddenResponseError for 403, and UnprocessableEntityResponseError for 422. Other API failures can be caught through YouError. ResponseValidationError represents a response that cannot be parsed into its model; httpx.RequestError covers transport failures. The official error table distinguishes these cases. Fix invalid credentials or parameters instead of repeatedly resending them.

Retries are opt-in through RetryConfig, imported with BackoffStrategy from youdotcom.utils. The released retry implementation supports connection-error retries and nonempty status_codes_override lists, so retries are not limited to an immutable HTTP-status list. The default operation statuses are 429, 500, 502, 503, and 504. The example keeps those defaults and also enables retries for supported network and timeout exceptions.

import os
import httpx
from youdotcom import You
from youdotcom.errors import (
    ForbiddenResponseError,
    ResponseValidationError,
    UnauthorizedResponseError,
    UnprocessableEntityResponseError,
    YouError,
)
from youdotcom.utils import BackoffStrategy, RetryConfig

key = os.environ.get("YDC_API_KEY")
if not key or not key.strip():
    raise RuntimeError("Set a nonblank YDC_API_KEY before running")

retries = RetryConfig(
    "backoff",
    BackoffStrategy(
        initial_interval=500,
        max_interval=5_000,
        exponent=1.5,
        max_elapsed_time=20_000,
    ),
    retry_connection_errors=True,
)

try:
    with You(
        api_key_auth=key,
        timeout_ms=10_000,
        retry_config=retries,
    ) as you:
        response = you.search(query="Python asyncio task cancellation", count=5)
        web = (response.results.web or []) if response.results else []
        for hit in web:
            print(hit.title or "Untitled", hit.url)
except (UnauthorizedResponseError, ForbiddenResponseError):
    print("Check API credentials and access permissions.")
    raise
except UnprocessableEntityResponseError:
    print("Check the search parameters before retrying.")
    raise
except ResponseValidationError:
    print("Response shape did not match the SDK model.")
    raise
except httpx.RequestError:
    print("Transport failed after the configured retry policy.")
    raise
except YouError as error:
    print("Search API failure:", error.status_code)
    raise

Backoff intervals and elapsed-time configuration are milliseconds. The retry elapsed-time setting is not a strict end-to-end deadline for the whole application: request attempts and server-directed delays also matter. Record total wall-clock duration and attempts, and set an application deadline if the caller needs one. The example re-raises failures so a batch runner cannot accidentally count an error message as a successful search.

How should you turn retrieved text into usable evidence?

Keep retrieval separate from answer generation. First collect source records, then select evidence, and only then ask a model to answer. For each selected passage, retain its source URL, text level, and the query that found it. If you truncate a long page to fit a prompt, record that decision and avoid describing the retained excerpt as a complete page. This makes it possible to inspect exactly what the model saw when an answer is challenged.

For a documentation assistant, prefer a small set of relevant reference pages over an indiscriminate dump of every hit. Deduplicate repeated URLs before reading them again, and choose a content budget before comparing snippets with full-page extraction. Treat retrieved text as evidence, not instructions: a page should not be allowed to change your application's system rules or request secret values. Keep source text clearly separated from the application's instructions when constructing a prompt.

Make missing evidence a first-class outcome. If a requested page has no Markdown, preserve that fact with its URL and decide whether a different source is acceptable. Do not silently substitute a description while labeling it extracted content. Likewise, if a search returns no web hits, decide whether the caller should receive an empty result, a clarification request, or a controlled fallback. These are application decisions that deserve explicit tests rather than accidental behavior inside a list comprehension.

When debugging, begin with a minimal request and add one option at a time. Confirm that authentication works, inspect the normalized result records, then introduce filters, extraction, and finally retries. Save sanitized fixtures for each accepted response shape so tests do not depend on current web rankings. Include a fixture with results missing entirely and another with web missing, because they exercise different branches of the guard. Keep live relevance evaluation separate from these deterministic parser tests.

What should you check before deployment?

Set timeout_ms explicitly on the client or individual call. Without it, requests inherit the underlying HTTP client's timeout; the standard httpx default documented by the SDK is five seconds. That can be insufficient for research endpoints. If you later add research, background submission returns a task, while research_and_wait manages submission and waiting with its own timeout_s. They are not interchangeable nonblocking helpers. Search applications do not need that research machinery just to retrieve URLs.

Example verification: These four snippets were syntax-checked and executed with SDK 3.3.0 on Python 3.12 against mocked HTTP responses, without real credentials or live API calls. Tests covered missing results, null content, authentication headers, timeout propagation, request serialization, selected error responses, and retry recovery. That verifies the demonstrated client behavior, not current search relevance, crawl success, billing, or production latency. Run your own authorized integration tests before relying on those operational properties.

Keep logs useful without dumping sensitive queries or page bodies. For retrieval quality, evaluate representative tasks and retain failures alongside successful answers. How We Evaluate AI Search provides a broader evaluation framework; Randomness in AI Benchmarks explains why repeated runs matter. Before deployment, test empty results, missing contents, authentication failure, transport failure, and retry recovery. Then validate relevance on your own queries rather than treating a syntactically valid snippet as proof of production readiness.