How to Build a LlamaIndex Web Search Retriever with the You.com Web Search API

TLDR: In LlamaIndex a Retriever is the abstraction query engines and workflows compose, so wrapping live web search as a Retriever makes every result a first-class node with a score. You subclass BaseRetriever, implement _retrieve to call the You.com Web Search API, and wrap each hit in a TextNode whose metadata carries the URL and page age. This guide covers the class, the composition into a query engine, when a Tool is the better wrapper, and the failure modes that turn an empty result set into a confabulation.

LlamaIndex gives you two ways to hand an external source to an agent stack, and picking the right one first saves a rewrite. A Tool is a function an agent chooses to call, which is the right shape when a language model is driving. A Retriever is a component query engines, routers, and workflows compose with directly, which is the right shape when your code is driving, when you want scored nodes flowing into synthesis, or when the same live-web source should serve both an engine and an agent. The Web Search API from You.com is the data source underneath, and the LlamaIndex retriever module guide is the upstream reference for the abstraction this article wraps it in.

What Is a Retriever in LlamaIndex?

A Retriever is the interface that turns a query into a list of scored nodes. BaseRetriever defines the contract, your subclass implements _retrieve, and the base class handles the synchronous and asynchronous wrappers around it. Each result is a NodeWithScore, a node plus a relevance score, which is the currency every downstream component understands: query engines synthesize over retrieved nodes, routers pick paths on them, and workflows pass them between steps.

The practical difference from a Tool is composability. A Tool hands a string to a model and gets a string back, which is perfect for agent loops and invisible to everything else. A Retriever's output is inspectable data with scores and metadata, which means citation lists, reranking, filtering, and node postprocessing all have something to grip. If your app needs the source URL of every claim the engine makes, the Retriever is the wrapper that keeps that URL alive from fetch to final answer.

How Do You Wrap the Web Search API as a Retriever?

The wrapper is one class and one helper. The helper calls the API with the standard library's urllib.request, the class turns each hit into a TextNode with the URL and page age in metadata and a monotonically decreasing score from its rank:

import json
import os
import urllib.request

from llama_index.core import QueryBundle
from llama_index.core.retrievers import BaseRetriever
from llama_index.core.schema import NodeWithScore, TextNode


def web_search(query: str, count: int = 5) -> list:
    req = urllib.request.Request(
        "https://ydc-index.io/v1/search",
        data=json.dumps({"query": query, "count": count}).encode(),
        headers={
            "Content-Type": "application/json",
            "X-API-Key": os.environ["YOUCOM_API_KEY"],
        },
        method="POST",
    )
    with urllib.request.urlopen(req, timeout=30) as resp:
        payload = json.load(resp)
    hits = payload.get("results", {}).get("web", []) or []
    return [
        {
            "url": h["url"],
            "title": h.get("title", ""),
            "text": (h.get("snippets") or [h.get("description", "")])[0],
            "page_age": h.get("page_age", ""),
        }
        for h in hits
    ]


class YoucomWebRetriever(BaseRetriever):
    def _retrieve(self, query_bundle: QueryBundle) -> list:
        nodes = []
        for rank, hit in enumerate(web_search(query_bundle.query_str), start=1):
            node = TextNode(
                text=hit["text"] or hit["url"],
                metadata={
                    "url": hit["url"],
                    "title": hit["title"],
                    "page_age": hit["page_age"],
                },
            )
            nodes.append(NodeWithScore(node=node, score=1.0 / rank))
        return nodes

Three decisions are load-bearing. The snippet falls back to the description field, and the text falls back to the URL, because a hit with no text is a data condition your engine should see as a thin node rather than a crash. The score is 1.0 divided by rank, a monotonic placeholder that preserves ordering without pretending search rank is a calibrated probability. And the URL and page age go into node metadata, not the text, so postprocessors can filter and citation logic can read them without parsing prose. The full request and response surface, including the count, domain, and freshness filters, is in the Web Search API guide, and the field-level reference is in What Is in a Web Search API JSON Response?

How Do You Compose It into an Engine or an Agent?

The engine path is one line plus your configured language model:

from llama_index.core.query_engine import RetrieverQueryEngine

retriever = YoucomWebRetriever()

engine = RetrieverQueryEngine.from_args(retriever)
answer = engine.query("What changed in the most recent Python patch release?")

for node in engine.retrieve(
    "What changed in the most recent Python patch release?"
):
    print(node.score, node.node.metadata["url"])

RetrieverQueryEngine synthesizes an answer over the retrieved nodes with the language model configured in your LlamaIndex settings, so the same retriever serves both the generation path and the inspection path above. The agent path reuses the same helper rather than the retriever: you wrap web_search as a Tool and register it with your agent workflow, which is exactly the pattern in our guide to adding a web search Tool to LlamaIndex agents. When a stack needs both, the helper is the single source of truth and the two wrappers stay thin.

What Are the Failure Modes?

Four account for most broken LlamaIndex integrations.

Empty-set confabulation is the serious one: the search returns nothing, the retriever returns an empty node list, and the query engine synthesizes an answer anyway from the language model's priors, which defeats the entire point of retrieving. Detection: assert a minimum node count at the application boundary and refuse or rephrase below it, because an engine asked to answer with zero nodes is a hallucination machine with extra steps. Import drift is the annoying one: LlamaIndex moved its core imports into the llama_index.core package in late 2023, and tutorials from before that era import from paths that no longer exist. Detection: pin the llamaindex package versions in your project and treat an ImportError as a docs check against your installed version, not a bug in your code. Metadata loss is the silent one: dropping the URL from node metadata is easy to do and nothing complains, but citation logic downstream now has nothing to cite and quietly stops emitting sources. Detection: a test that retrieves one query and asserts url in every node's metadata. Score abuse is the subtle one: rerankers and fusion strategies consume scores as comparable numbers, and 1.0 divided by rank is an ordinal convenience, not a calibrated likelihood. Detection: if you attach a reranker, read its docs on expected score semantics before trusting the blend.

Related Guides

FAQ

What is the difference between a Retriever and a Tool in LlamaIndex? A Tool is a function an agent's language model chooses to call, returning a string. A Retriever is a component query engines and workflows compose with, returning scored nodes with metadata. Choose the Retriever when code drives retrieval or you need inspectable nodes, and the Tool when an agent drives.

Which method do I implement on BaseRetriever? _retrieve, taking a QueryBundle and returning a list of NodeWithScore. The base class wraps your implementation with the synchronous and asynchronous entry points. Check the retriever module guide for your installed version, since the package reorganized its imports in late 2023.

Why put the URL in node metadata instead of the text? Metadata survives composition. Postprocessors filter on it, citation logic reads it without parsing prose, and the text stays clean for the language model. A URL buried in the text is lost the moment anything rewrites the node text.

What happens when the web search returns no results? The retriever returns an empty node list, and the failure depends on what you do next. A query engine will synthesize from priors, which is a confabulation. Assert a minimum node count at your application boundary and refuse or rephrase below it.

Frequently Asked Questions

A Tool is a function an agent's language model chooses to call, returning a string. A Retriever is a component query engines and workflows compose with, returning scored nodes with metadata. Choose the Retriever when code drives retrieval or you need inspectable nodes, and the Tool when an agent drives.

_retrieve, taking a QueryBundle and returning a list of NodeWithScore. The base class wraps your implementation with the synchronous and asynchronous entry points. Check the retriever module guide for your installed version, since the package reorganized its imports in late 2023.

Metadata survives composition. Postprocessors filter on it, citation logic reads it without parsing prose, and the text stays clean for the language model. A URL buried in the text is lost the moment anything rewrites the node text.

The retriever returns an empty node list, and the failure depends on what you do next. A query engine will synthesize from priors, which is a confabulation. Assert a minimum node count at your application boundary and refuse or rephrase below it.