Web content extraction API for AI agents Contents API

A web content extraction API takes a URL and returns the readable content of that page as structured data, usually Markdown or HTML, with the navigation, ads, cookie banners and footers removed. Developers use one instead of running their own headless browsers and parsers. For AI teams, it is the step that turns a list of links into text a model can read, whether that is for a RAG index, a monitoring job, or an agent that needs to read a page before it acts.

It is a URL to Markdown API with HTML and metadata on request. You send up to 10 URLs to POST https://ydc-index.io/v1/contents and get back a structured object for each one: the URL, the page title, clean Markdown and/or HTML, and optional metadata such as site name and favicon. Pages are crawled in parallel, JavaScript-rendered pages are rendered first, and PDFs are converted to Markdown. There is no parsing or browser automation for you to maintain.

Both return full page content; they start from different places. Use the Contents API when you already know the URLs you want to read. Use the Web Search API with extraction_mode set to full_page when you have a query and want You.com to discover the pages and return their content in the same call. Contents handles 10 URLs per request at $1.00 per 1,000 pages. Web Search returns up to 100 results per search at $5.00 per 1,000 searches, plus $1.00 per 1,000 pages crawled live.

Yes to both. Single-page apps, dashboards and other script-heavy pages are rendered before extraction; use crawl_timeout to give them time to load, with 5 to 10 seconds for static pages and 20 to 30 seconds for JavaScript-heavy ones, up to a maximum of 60. PDF URLs are parsed and returned as Markdown like any other page. If a single-URL request points at a PDF that is too long to process, the API returns a 422 with the code pdf_too_large; in a multi-URL request that URL is left out of the 200 response so the rest of the batch still succeeds.

The do-it-yourself path works for simple static pages and breaks everywhere else: client-rendered apps need a headless browser, PDFs need separate tooling, boilerplate rules drift as sites change, and you end up maintaining retries, timeouts, caching and concurrency. The Contents API handles rendering, boilerplate removal, PDF parsing, caching and parallel batching on the server, returns the same clean Markdown or HTML for every URL, and costs $1.00 per 1,000 pages, so your team's time goes into the pipeline, not the parser.

You choose with the formats parameter: markdown, html, metadata, or any combination. Markdown is the default and the best fit for prompts and vector stores because boilerplate is removed. HTML gives you the raw document for your own parsing. Metadata adds site_name and favicon_url for source attribution in a UI. Each format adds processing time, so request only what you need.

The Contents API may return cached page content to improve latency. The max_age parameter, in seconds, controls how fresh that cache must be. For example, max_age=86400 ensures content is no older than 24 hours, and max_age=0 bypasses the cache and re-fetches the page on every request. Leave it unset to accept cached content regardless of age.

Up to 10 URLs per request, crawled in parallel. One request with 10 URLs is faster than 10 separate requests. For larger jobs, chunk your list into batches of 10. If a URL in a batch can't be crawled, for example because it is behind a login wall or returns a 404, its markdown and html fields come back as null and the other URLs are unaffected.

Yes. Knowledge base ingestion is one of the core use cases. Pass your list of authoritative sources such as documentation pages, whitepapers or wikis, convert them to clean Markdown with metadata, and index the results into your vector store. Markdown without navigation, ads and footers is far smaller than raw HTML, so it costs fewer tokens to embed or prompt with, and its headings give you natural chunk boundaries. The same pattern works for a research assistant that reads a user-supplied URL on the fly and feeds it to an LLM as context.

It extracts the readable content of public web pages, which is what most teams mean by scraping for AI. Competitive intelligence is a common use: fetch competitor pricing, feature or blog pages on a schedule, feed the Markdown to an LLM, and surface meaningful changes. Use max_age to control freshness and crawl_timeout for pages that render client-side. You are responsible for respecting each site's terms of use.

The Contents API fetches publicly accessible pages. A URL behind a login wall, a paywall or a hard block can't be crawled; in that case its markdown and html fields come back as null and the rest of the batch is unaffected, so check for null before processing. It does not bypass authentication or anti-bot systems, and you are responsible for respecting each site's terms of use. For pages that merely load slowly or render client-side, raise crawl_timeout rather than assuming a block.

The youdotcom Python SDK exposes you.contents(urls=[...], formats=[...]). Any REST client works too; send your key in the X-API-Key header. Agents and IDEs can use the You.com MCP server at api.you.com/mcp, where page fetching is available as a tool alongside web search, so Claude, Cursor and other MCP clients can read pages without custom glue code.

The Contents API costs $1.00 per 1,000 pages returned, with Markdown, HTML and metadata all included in that one price. Pricing is pay as you go with no minimums, seats or platform fee, and volume discounts and annual pricing are available through sales. New accounts get $100 in free credits with no credit card required, which covers roughly 100,000 pages: enough to prototype an ingestion pipeline, run a monitoring job for weeks, or evaluate extraction quality on your own URLs before paying anything.

All of these turn URLs into model-ready text. The differences that matter are batch size, rendering, freshness control and price. The You.com Contents API takes 10 URLs per request in parallel, renders JavaScript, parses PDFs, lets you set cache freshness with max_age, and costs $1.00 per 1,000 pages with Markdown, HTML and metadata included in one price. It also shares an API key, SDK and billing with the You.com Web Search, Answer and Research APIs, so discovery and reading live in one stack.

No. The URLs you send and the content returned are never used for model training. Zero Data Retention (ZDR) is available for enterprise agreements and automatically purges request and response content. The platform is SOC 2 certified and DPA-ready.