How to Build a Breach Monitoring Pipeline with the You.com Web Search and Contents APIs

TLDR: Breach monitoring means detecting newly disclosed security incidents that affect your organization, your vendors, or your customers, as early as public information allows. A working pipeline runs news searches per monitored entity on a schedule, deduplicates what it has seen, extracts the details from promising articles, and routes verified incidents to a human. This guide builds that pipeline with the You.com Web Search and Contents APIs, gives the decision framework for when news-based monitoring is the right layer, and names the failure modes that generate false alarms.

Breach news breaks on the web first. Incidents surface through disclosure statements, security reporting, and regulatory filings before they land in any structured feed. You.com does not sell a breach monitoring product, and this guide does not describe one. It describes how to build the pattern from retrieval primitives: the Web Search API for live news discovery and the Contents API for extracting what an article actually says.

What Are You Actually Monitoring?

A breach pipeline watches the public disclosure cycle, which has a reliable order: an incident happens, a disclosure follows, security press covers it, a vendor or regulator publishes an advisory, and months later structured datasets absorb it. Monitoring the news layer of that cycle catches incidents hours to days earlier than feeds built on the last stage. Complementary layers cover different objects: credential dump monitoring services such as Have I Been Pwned, whose API documents per-account breach queries, answer whether specific credentials appeared in known dumps, while vulnerability feeds answer which software you run is affected. Those services do what they do well. The gap a news pipeline fills is the ambiguous middle: vendor incidents, supplier exposure, and brand mentions that no structured feed will ever name.

How Do You Run the Monitoring Loop?

The loop is one search call per monitored entity per run, restricted to news and recent time. The Web Search API returns ranked web and news results with URLs, snippets, and page age, and accepts a freshness filter, per its guide. The query pattern matters more than cleverness: entity name plus a small set of incident terms, freshness clamped to the run cadence.

import json
import os
import urllib.request

def news_search(entity: str, count: int = 10) -> list:
    key = os.environ["YOUCOM_API_KEY"]
    query = entity + " data breach OR leak OR exposed OR incident"
    req = urllib.request.Request(
        "https://ydc-index.io/v1/search",
        data=json.dumps({"query": query, "count": count,
                         "freshness": "week"}).encode(),
        headers={"Content-Type": "application/json", "X-API-Key": key},
        method="POST",
    )
    with urllib.request.urlopen(req, timeout=30) as resp:
        payload = json.load(resp)
    results = payload.get("results", {})
    return results.get("news", []) or results.get("web", [])

def run_cycle(entities: list, seen: set) -> list:
    alerts = []
    for entity in entities:
        for hit in news_search(entity):
            if hit["url"] in seen:
                continue
            seen.add(hit["url"])
            alerts.append({"entity": entity, "url": hit["url"],
                           "title": hit.get("title", ""),
                           "published": hit.get("page_age", "")})
    return alerts

watchlist = ["Example Corp", "Example Vendor Inc"]
seen_urls = set()
for alert in run_cycle(watchlist, seen_urls):
    print(alert["entity"], "|", alert["published"], "|", alert["title"])

Persist seen_urls across runs, in a file or a table, because dedupe by URL is what stops the pipeline from re-alerting the same story every cycle. The second half of the pipeline is extraction: for each alert, the Contents API fetches the article's full content as markdown, per its guide, so your triage step can read entity names, incident dates, and data types instead of a headline's guess at them.

When Is News-Based Monitoring the Right Layer?

The decision framework turns on three questions. First, what are you protecting: if the answer is specific employee credentials, a breach-data service with per-account queries is the correct primary layer and news monitoring is a supplement. Second, how large is the perimeter: a dozen entities is a cron job, thousands of vendors is a data engineering problem that needs batching and a queue, regardless of API. Third, what does an alert trigger: if the response plan requires confirmed details, budget for the extraction and human triage step, because news alerts alone are leads, not findings. Security teams that skip the third question end up with an alert queue nobody trusts.

What Failure Modes Generate False Alarms?

Each failure mode has a detection you can build. Anniversary re-reporting: outlets re-cover old breaches on their anniversaries, and a naive pipeline re-alerts a three-year-old incident. Detection: require the article's page age inside your freshness window and extract the incident date, then drop alerts whose incident predates the window. Name collisions: a common company name matches every small business with the same name that suffers an incident. Detection: add disambiguating terms to the query, such as the domain or industry, and require corroboration across at least two distinct sources before an alert reaches a human. Speculative coverage: journalists write about breach risk and post-mortems that contain the words but describe no incident. Detection: the extraction step should look for the markers of a confirmed disclosure, a named party, a date, a data type, and suppress stories lacking them. Paywalled or bot-walled articles: extraction returns partial text and the triage step judges on a fragment. Detection: check the extracted length and flag thin extractions for manual reading rather than auto-classifying them.

The pipeline's output is evidence for a human analyst, not a verdict. That framing keeps monitoring honest, the same discipline our LLM citations guide applies to generated claims generally.

Related Guides

FAQ

Is there a You.com breach monitoring API? No. You.com provides retrieval primitives: the Web Search API for live news discovery and the Contents API for extracting article content. A breach monitoring pipeline is something you build on top of those, which is what this guide shows.

Can this pipeline monitor the dark web? No. Live web search surfaces publicly published pages, and dark web content sits outside that index. If you need dark web coverage, that is a specialized vendor capability. What a news pipeline catches is the public disclosure cycle, which is where most organizational exposure first becomes visible anyway.

How often should the monitoring loop run? Match the cadence to your response SLA. A daily run covers most vendor and brand monitoring cases. If a security operations team needs same-day triage, run every few hours and clamp freshness to the interval, accepting that higher cadence multiplies your API volume.

How do I stop the pipeline from alerting on the same story twice? Dedupe on two keys: URL, which catches re-alerting the same article, and the extracted incident date plus entity, which catches multiple outlets covering one incident. Persist both across runs so each cycle only alerts on what is genuinely new.

What belongs in the triage step after extraction? Three checks: the incident date is inside your window, the named entity actually matches your monitored entity and not a namesake, and the story describes a confirmed disclosure rather than speculation. Alerts passing all three go to a human with the extracted text attached, so the analyst never has to re-fetch the article.