How to Scrape Search Results With an API in 2026

TLDR: Getting search results as data in 2026 means choosing between three routes: scraping the results pages yourself, which fights blocks and terms of service, buying a SERP scraping API that does the scraping for you, or calling a web search API that returns clean results directly. This guide covers what changed in 2026, what each route costs you in reliability and maintenance, a working web search API example, and how to pick by use case.

The question used to have two answers: write a scraper, or buy a scraping API. Then 2026 arrived and the ground shifted. Microsoft retired the public Bing Search APIs on August 11, 2025, per Microsoft's own documentation, which removed the most common official endpoint for programmatic search results and pushed thousands of integrations onto alternatives. Google's Custom Search JSON API still exists with its documented quotas on the Google developer docs. Meanwhile, AI applications created a new default: most teams that need search results as data today are feeding models, not rendering result pages. If you are deciding now, You.com and others offer web search APIs that return the results directly, and the case for each route depends on what you are building.

Why Does Direct Scraping Keep Failing?

Scraping a search engine results page means automating a browser, rotating IP addresses, solving CAPTCHAs, and re-fixing your parser every time the page layout moves. It fails on four fronts.

Blocks. Search engines actively detect and rate-limit automated access. A scraper that works Monday returns CAPTCHAs by Friday, and the countermeasures escalate continuously.
Terms. Google's Terms of Service prohibit automated access without permission, as published at policies.google.com/terms. Whether or not you ever see consequences, shipping a product on a foundation the source explicitly disallows is a business risk, not just an engineering one.
Maintenance. Result pages are optimized for humans and change without notice. Every layout shift silently corrupts your parsed fields until someone notices the data is wrong.
Cost shape. Proxies, solver services, and headless browser infrastructure all bill monthly whether or not the data arrives clean.

The failure mode to watch for is silent data corruption: your scraper keeps returning 200 responses with half-empty fields, and the downstream dataset degrades for weeks before anyone measures it. Detection means validating field completeness on every parse, not just checking status codes.

What Do SERP Scraping APIs Do for You?

A SERP scraping API is a productized version of the scraper: the vendor runs the proxies, solves the blocks, maintains the parsers, and returns the search engine results page as JSON. SerpApi, for example, describes its product on its homepage as a Google Search API that returns SERP data as JSON, as of September 2026. The category exists because the maintenance burden above is real, and outsourcing it is a rational buy.

What you get back is SERP-shaped: organic positions, ads, related searches, and the other page furniture of a results page. That is exactly what SEO rank tracking needs, and it is more than an agent or RAG pipeline needs. The tradeoffs are the per-request pricing model, which scales with your query volume, and the fact that you are still receiving a mirror of a page designed for something other than your use case. Our SERP API guide covers this category in depth, and the deeper conceptual comparison is in our SERP scraping API vs web search API walkthrough.

How Do You Get Clean Results With a Web Search API?

A web search API skips the results page entirely. You send a query, you get structured results: url, title, description, and snippets per item, with metadata like publication dates. No proxies, no parsers, no page furniture. The You.com Web Search API does this from one endpoint.

import json, os, urllib.request

def search(query: str, count: int = 5) -> list:
    req = urllib.request.Request(
        "https://ydc-index.io/v1/search",
        data=json.dumps({"query": query, "count": count}).encode(),
        headers={
            "X-API-Key": os.environ["YDC_API_KEY"],
            "Content-Type": "application/json",
        },
        method="POST",
    )
    try:
        with urllib.request.urlopen(req, timeout=15) as resp:
            data = json.loads(resp.read().decode())
    except urllib.error.HTTPError as e:
        return [{"error": f"search failed with HTTP {e.code}"}]
    return (data.get("results") or {}).get("web", [])

The response carries a web array of results and, when the query has news intent, a news array alongside it. Filters matter for this use case: freshness windows such as day, week, or month keep results current, and domain steering lets you restrict or exclude sources. The JSON response guide walks the full field structure.

Which Approach Fits Which Job?

The decision splits on one question: do you need the search engine's page, or just its answers?

SEO rank tracking and SERP monitoring: you need positions, ads, and related queries, which is the SERP scraping API's native shape. A web search API deliberately does not model rank positions.
Agent grounding, RAG, and answer pipelines: you need clean results with snippets and dates, not page furniture. A web search API is built for exactly this, and it is the category the You.com Web Search API belongs to.
Migrating off the retired Bing APIs: both routes are candidates, but match on intent. If your old integration consumed clean search results, a web search API is the like-for-like replacement. Our Bing Search API alternative guide covers the migration in detail.
Light, free-tier usage: Google Custom Search JSON API remains an official documented option with published quotas, covered in our Google CSE guide.

What Should You Monitor After You Switch?

Whichever route you choose, three failure modes recur.

Stale cache. Scraping APIs serve cached SERP snapshots, and a cache that is hours old is fine for rank tracking and wrong for freshness-sensitive pipelines. Detection: query a known-fresh term and compare the result timestamp to reality.

Zero-result queries. Every provider occasionally returns empty results for niche queries. Detection: track the zero-result rate per provider and per query family, and alert when it jumps.

Quota and billing surprises. Per-request pricing plus an agent that searches before every answer is how a prototype becomes a bill. Detection: alert on request volume by key, and cap per-task search counts in your agent loop.

For a structured way to evaluate any of these options before committing, our search API evaluation guide gives you the test harness.

Related Guides

FAQ

Is scraping Google search results legal? This guide is engineering guidance, not legal advice. The practical facts: Google's terms of service prohibit automated access without permission, and the technical countermeasures exist regardless of the legal question. Teams that need certainty should ask their own counsel before building on scraping.

What replaced the Bing Search API? Nothing official from Microsoft. The retirement on August 11, 2025 left integrations to move to SERP scraping APIs, Google Custom Search JSON API, or web search APIs, depending on which parts of the old contract they depended on. Our Bing alternative guide maps the migration options.

Can a web search API track my SEO rankings? Not directly. Web search APIs return clean results for queries, not rank positions for your domain over time. Rank tracking wants the SERP's own positions, which is the SERP scraping API's job.

Which is cheaper at volume? It depends on your query pattern and the vendors you compare, and pricing claims need current numbers rather than blog-aging ones. The pricing pages of the vendors on your shortlist, including You.com's own, are the source to check before you commit.

LI Test
LI Test

Related resources.

No items found.