What Is a Web Content Extraction API? How to Build a Pipeline With the You.com Contents API

TLDR: A web content extraction pipeline turns scattered pages into clean Markdown your models can use. The two moves that matter: use the You.com Web Search API to discover candidate URLs from a query, and the Contents API to read chosen URLs deeply and on a schedule. This guide shows both patterns, the failure modes that bite in production, and the decision rules for when to use each.

Why Not Just Scrape the Pages Yourself?

Writing a scraper is easy. Operating one for months is not. A hand-rolled pipeline needs a headless browser for JavaScript-rendered pages, per-site parsing rules that break on every redesign, proxy rotation for hosts that block datacenter traffic, and a blocklist maintenance habit that nobody enjoys. Each of those is a real engineering cost you pay forever.

The pattern that survives is a build-vs-buy decision with three named tradeoffs. Control: your own scraper gives you total control over extraction logic, an extraction API gives you a stable interface instead. Coverage: DIY scrapers only reach pages you already know, while search-integrated extraction discovers pages you did not. Maintenance: parsing rules rot silently as sites change, while an extraction service's job is to keep up with that rot so you do not. If your differentiator is the extraction logic itself, build. If your differentiator is what you do with the content, buy.

How Do You Extract Content From URLs You Already Know?

The Contents API is the tool for known URLs. Pass up to 10 URLs in a single request (You.com Contents guide, 2026-09-04), choose your formats from markdown, html, and metadata, and the API crawls them in parallel and returns clean page content for each. Markdown is the default and strips navigation menus, ads, and footers, so the output drops straight into a prompt or a chunker.

Here is a working knowledge-base ingestion pattern with realistic failure handling.

from youdotcom import You
from youdotcom.models import ContentsFormats

SOURCE_URLS = [
    "https://docs.example.com/api-reference",
    "https://docs.example.com/authentication",
    "https://docs.example.com/rate-limits",
]

def ingest(urls):
    documents = []
    with You() as you:
        pages = you.contents(
            urls=urls,
            formats=[ContentsFormats.MARKDOWN, ContentsFormats.METADATA],
        )
        for page in pages:
            if not page.markdown:
                # Login wall, 404, or blocked crawl: fields come back null
                print(f"skipping (no content): {page.url}")
                continue
            documents.append({
                "source": page.url,
                "title": page.title,
                "content": page.markdown,
            })
    return documents

docs = ingest(SOURCE_URLS)
print(f"ingested {len(docs)} of {len(SOURCE_URLS)} pages")
# Next step: chunk each doc and index into your vector store

Two parameters change real-world outcomes. crawl_timeout, which runs 1 to 60 seconds with a default of 10, decides whether JavaScript-heavy pages finish rendering before the crawl gives up. When raising crawl_timeout above the default, also set timeout_ms on the You() constructor to a value larger than crawl_timeout in milliseconds; the SDK's default HTTP timeout is 5 seconds and will fire before the server responds on slow pages (You.com SDK docs, 2026-09-04). And max_age, set in seconds, controls cache freshness: a monitoring job that must catch page changes should set a max_age matching its freshness requirement, because a stale cache hit looks exactly like an unchanged page.

How Do You Discover Pages and Extract Them in One Step?

When you start from a query rather than a URL list, use full page extraction inside the Web Search API. Add the extraction object to a POST /v1/search request with extraction_mode set to full_page, and every result gains a contents object with the whole page. Set extraction_formats to markdown, which is the default for full_page extraction.

This is the discovery pattern: one call returns results across the web and news sections, up to 100 results per search, with count defaulting to 10 per section (You.com Contents guide and Search API reference, 2026-09-04), with full content attached to each.

from youdotcom import You
from youdotcom.models import Extraction, ExtractionFormat, ExtractionMode

with You() as you:
    res = you.search(
        query="llm evaluation benchmarks 2026 filetype:pdf",
        count=10,
        extraction=Extraction(
            extraction_mode=ExtractionMode.FULL_PAGE,
            full_page={"extraction_formats": [ExtractionFormat.MARKDOWN]},
        ),
    )

    if res.results and res.results.web:
        for result in res.results.web:
            if result.contents and result.contents.markdown:
                print(result.url, len(result.contents.markdown))
            else:
                print(f"no content extracted: {result.url}")
    else:
        print("no results, handle this case explicitly")

The query supports search operators, so filetype:pdf, site:domain.com, and boolean terms work for narrowing discovery before you pay to crawl. For heavy domain filtering, the documentation recommends POST requests, where domain lists go in as plain JSON arrays.

When Do You Use Each Pattern?

One rule covers it: discovery and re-reading are different jobs.

Use search with full_page extraction when the page set is unknown and dynamic, such as collecting fresh coverage of a topic or filling a RAG context from the open web. Use the Contents API when the page set is known and stable, such as monitoring competitor pricing pages or re-ingesting a documentation index on a schedule. Most mature pipelines end up running both: search discovers the candidate set once, a human or a filter picks the winners, and the Contents API re-reads that list on a cadence with max_age set to the freshness the job actually needs.

How Do You Keep Extraction Costs in Check?

Extraction is metered work, so three habits keep a pipeline from overpaying. First, extract snippets when snippets suffice: a default search call returns title and snippet per result at lower cost than full_page extraction, and for many ranking and filtering jobs the snippet is enough to decide which pages deserve a full read. Second, cache on your side before you re-fetch: when you re-read the same URL list on a schedule, store the last content hash per URL and only process pages whose content actually changed, which is also the core of any monitoring job. Third, narrow the crawl before you widen it: use query operators and domain filters during discovery so the extraction budget lands on pages that matter rather than on the long tail of the SERP.

What Failure Modes Should You Plan For?

Four failure modes cover most production incidents in extraction pipelines.

The silent null. Pages behind login walls, 404s, and crawler-blocking hosts return null content fields while the rest of the batch succeeds. Detect it by checking page.markdown before processing, every time. A null that reaches your chunker becomes a runtime crash discovered by your users.

The empty result set. A query that matches nothing returns empty results arrays. The documentation is explicit: always check whether results.web or results.news are empty before processing. Detection is a one-line guard. The failure without it is a confidently empty answer downstream.

The incomplete render. JavaScript-heavy pages can exceed the crawl timeout and come back partial or null. Detection is repeated nulls or suspiciously short output on known-long pages. The fix is raising crawl_timeout toward the 60-second ceiling for those specific targets.

The boilerplate leak. Even clean Markdown extraction can occasionally carry navigation remnants from unusual layouts. Detection is spot-checking output on any new domain before scaling it into your index. A page that looks clean in a browser can still produce noisy Markdown.

Where Does This Pipeline Fit?

Extraction is the middle of the stack. Search and discovery sit upstream, chunking, indexing, and generation sit downstream. The CommonMark specification is the reference for the Markdown most extraction APIs emit, and OpenAI's token counting guide is a practical primer on why text volume, not page count, is the unit that matters downstream.

For the adjacent workflows, see the Contents API hub article for the product detail, news API pipelines for monitoring flows, and B2B data workflows and lead enrichment for enrichment patterns built on the same primitives. Current rates for both APIs are on the You.com pricing page.

Next action: get a key from the You.com platform, list 10 URLs your team checks by hand every week, and run the ingest function above against them. Whatever comes back clean is time you stop spending on manual copy-paste, starting today.