What Is the You.com Contents API? Clean Page Content From Any URL

TLDR: The You.com Contents API extracts clean, ready-to-use page content from URLs you choose. Pass up to 10 URLs in one request and get Markdown or HTML back for each. No headless browser, no HTML parsing, no boilerplate cleanup. This guide covers the parameters that matter, the failure modes to handle, and when to reach for it instead of full page extraction inside a search call.

What Does the You.com Contents API Do?

The You.com Contents API takes a list of URLs you already know and returns the full content of each page as clean Markdown or HTML, structured for LLM consumption. You provide the URLs and the API crawls them in parallel and hands back page content with no navigation menus, ads, or footers attached.

Each URL in your request returns a structured object with the source URL, a page title, the content in each format you requested, and optional site metadata. You control the output through the formats parameter and can request markdown, html, and metadata in any combination. Markdown is the default and the right choice for feeding a model. The metadata format adds the site name and favicon URL, which is useful when you are building a UI that shows source attribution.

You pass up to 10 URLs in a single request. One request with 10 URLs is faster than 10 separate requests because the API processes the batch in parallel, so batching is the default habit to build.

Contents API or Web Search API With Full Page Extraction: Which Fits Your Workflow?

It's easy to choose if you start by asking one question: do you start with URLs or with a query?

The Contents API and full page extraction in the Web Search API both return full page content, but they serve different workflows. The Contents API starts from URLs you already have, such as a competitor pricing page list or a documentation index you maintain. The Web Search API with extraction_mode set to full_page starts from a query and returns up to 100 search results with content attached in a single call.

The tradeoff is control versus discovery. With the Contents API you choose exactly which pages get read, which matters for monitoring jobs where the target list is stable and known. With search-first extraction you get discovery for free, but the page set is whatever the search returns. Many pipelines use both: search to discover candidate pages once, then the Contents API to re-read the winners on a schedule.

There is a cost shape difference as well. Search-plus-extraction is billed as a search call plus pages crawled, while the Contents API is billed per page fetched. The exact rates are on the You.com pricing page.

How Do You Call the Contents API From Python?

The official youdotcom SDK wraps the request in one method call. Here is a working monitoring example that fetches three known pages and handles partial failure, which is the failure mode you will hit first in production.

from youdotcom import You
from youdotcom.models import ContentsFormats

COMPETITOR_PAGES = [
    "https://competitor-a.com/pricing",
    "https://competitor-b.com/pricing",
    "https://competitor-c.com/features",
]

with You() as you:
    pages = you.contents(
        urls=COMPETITOR_PAGES,
        formats=[ContentsFormats.MARKDOWN],
        crawl_timeout=15,
    )

for page in pages:
    if page.markdown:
        print(f"OK: {page.title} ({len(page.markdown)} chars)")
    else:
        print(f"FAILED: {page.url}")

The null check is not decorative. If one URL in a batch fails to crawl, for example a page behind a login wall or a 404, the API returns null for that page's markdown and html fields while the rest of the batch succeeds. Skipping the check means feeding None into your downstream pipeline and finding out from a stack trace instead of a log line.

Which Parameters Change the Outcome?

Three parameters do most of the work in real deployments.

formats: Request only what you need. Every extra format adds processing time. If a model is the consumer, request markdown alone and skip html. If a UI needs source attribution, add metadata for the site name and favicon.

crawl_timeout: The per-URL timeout in seconds, from 1 to 60, with a default of 10. Static pages are usually done in 5 to 10 seconds. JavaScript-heavy pages, such as single-page app dashboards, often need 20 to 30 seconds for the renderer to finish. If you see nulls on pages you know exist, a short timeout is the first suspect.

max_age: Cache freshness control in seconds. By default the API may return cached page content to improve latency. Set max_age to a threshold in seconds and any cached copy older than that is ignored, forcing a fresh fetch. For a daily monitoring job, max_age=86400 guarantees content no older than 24 hours. max_age=0 always bypasses the cache. Leave it unset when freshness does not matter and you want the fastest response.

What Failure Modes Should You Handle?

Four failure modes cover most production incidents with any extraction API, including this one.

The null crawl. Login walls, 404s, and pages that block crawlers return null content fields. Detection is a one-line check on page.markdown before processing. Batch jobs should log the failed URL and continue, not abort the batch.

The stale cache hit. A monitoring job that reads cached content can miss a page change entirely. If your job's purpose is detecting change, set max_age to your real freshness requirement instead of accepting the default cache behavior.

The slow render. A JavaScript-rendered page that exceeds crawl_timeout comes back incomplete or null. Detection is repeated nulls or suspiciously short content on known-long pages. The fix is a higher crawl_timeout for those targets, up to the 60 second maximum.

The boilerplate survivor. Markdown extraction strips most chrome, but unusual layouts can leak navigation or cookie-banner text into the output. Detection is spot-checking extracted content on new domains before you index them at scale.

What Should You Build With It?

Three patterns from the Contents API documentation cover most use cases.

Competitive intelligence: Fetch competitor pricing, feature, and blog pages on a schedule, feed the Markdown to an LLM, and surface meaningful changes without anyone checking pages by hand.

Knowledge base ingestion: You have a list of authoritative sources, such as documentation pages, whitepapers, and internal wikis. Fetch them all as clean Markdown and index them into your vector store. The Markdown output follows the CommonMark specification, so any Markdown-aware chunker works without custom parsing.

Research assistant: Let users ask questions about specific URLs. Fetch the page content on the fly and pass it as context to your LLM, which turns any URL into a searchable document.

Where Does It Fit in the Rest of the Stack?

The Contents API is one layer in a retrieval pipeline, not the whole pipeline. Use the Web Search API to discover pages from a query. Use the Contents API to read known URLs deeply and on a schedule. Use the Research API when you want the synthesis step done for you and a cited answer back. The CommonMark specification defines the Markdown flavor to expect in the output, and the W3C HTML standard pages are the reference when you request raw html instead.

Sibling articles in this hub cover the adjacent workflows: news monitoring pipelines, B2B data workflows, MAP violation monitoring, and lead enrichment.

What Should You Check Before You Scale?

Two checks before you point a batch job at thousands of URLs.

First, run a 10-URL sample from each domain you plan to ingest and read the Markdown by hand. Boilerplate leaks and truncated renders are obvious in a sample and invisible in an aggregate. Second, decide your max_age policy per job rather than per pipeline. A daily competitor price check and a weekly documentation re-index have different freshness requirements, and setting the strictest max_age everywhere trades latency you paid for without needing it.

Next action: get an API key from the You.com platform, run the Python example above against three URLs you care about, and check what comes back. The quickest way to feel the value is to point it at a page you have tried to scrape by hand.