6 Lessons We’ve Learned While Building A Search Engine for Developers
Fast code search is a superpower. But coding searches are hard and time-consuming, often requiring 85% more edits to queries than noncoding searches [1]. That’s why we’ve decided to build a search engine for developers. As you might expect, there were numerous issues we’ve dealt with when building a complex niche search engine; along the way, we learned a few lessons.
The best products are born from painful experiences
Developers use many different search methods to find what they need. They create new queries and modify existing ones, append sites to search results, use filters, skim through indexed documentation, and swap search terms until they find the file type they need. If they persist in this cycle of query creation and modification, eventually, they find what they need. Every click and every site load is an inefficiency that takes time away from solving coding problems and completing coding tasks.
Our community of developers who were living in this painful code search cycle would often ask why there wasn’t one easy-to-use search engine drawing from all the best coding resources.
Even Paul Graham wondered about this way back in 2012 and suggested that someone should “build the search engine that all the hackers use [2].”
Here’s an excerpt from Graham’s 2012 essay:
The way to win here is to build the search engine all the hackers use. A search engine whose users consisted of the top 10,000 hackers and no one else would be in a very powerful position despite its small size, just as Google was when it was that search engine. And for the first time in over a decade, the idea of switching seems thinkable to me.
Since anyone capable of starting this company is one of those 10,000 hackers, the route is at least straightforward: make the search engine you yourself want. Feel free to make it excessively hackerish. Make it really good for code search, for example. Would you like search queries to be Turing complete? Anything that gets you those 10,000 users is ipso facto good.
Since we already started an ambitious quest to change the internet forever with You.com, we decided to build a search engine for developers by developers — from scratch.
It’s how YouCode, the best search engine for coders, was born.
Tl;dr — summarization with AI
Mainstream search engines are not optimized for code search as the task of building one is complicated and difficult. Small coding sites either are not indexed by the major search engines or do not have enough traffic to warrant organization and maintenance. Big coding repositories containing clean code snippets can be overwhelming and painful to navigate, slowing down developers who are in a rush to find what they need.
We used artificial intelligence and natural language processing to parse large informational repositories into concise summaries with side panels that expand to reveal the most important information, along with code snippets that users can copy/paste. We iterated on the design and layout together with our community and introduced a horizontal scroll for each source app that summarizes information by topic, popularity, and subject for faster skimming. On YouCode, developers can search both vertically and horizontally to cover more ground faster.
In data we trust
Traditionally, developers have relied on Google for code retrieval. However, the first page of Google is pay-to-play real estate, where results are often misleading SEO content, ads, or tools that don’t have the right level of depth. This is inefficient for long-tail queries where the intent is not so obvious.
To solve for relevancy, we developed our own ranking system. It’s more sophisticated and relies on our own data processing and engineering to crawl, gather, and parse data in a more precise and careful way to provide the most relevant information on every search.
Intention vs. the intended action
Developer search queries are often more unique and more problematic than general consumer queries. It is often hard to know exactly what the right query will be to find the right document. It is fairly common that the complete answer doesn’t even exist. This is part of why developers often need to resort to trial and error rephrasing related searches to surface good examples. It’s a time sink — getting caught up in understanding different sites and samples instead of staying focused on completing the project.
YouCode aggregates 20+ best developer sources, including StackOverflow, GitHub, TowardsDataScience, Web3 Schools, and HuggingFace presented as apps that developers can prioritize (or omit) within their search results. This allows developers to tailor their search results to exactly what they need at a given moment, and the customization options will grow in the coming months.
There aren’t any ads on YouCode, and we don’t crawl any wonky SEO’d content, to help developers stay focused.
A search engine that codes for you
Coders also told us that sometimes they experience writer’s block when they are faced with new frameworks, coding languages, or bugs they cannot fix. To help combat this, we developed Code Complete, an AI code completion tool that writes code within search results, saving time and helping developers code faster.
Code Complete is a powerful code suggestion tool that has been trained on billions of lines of code from thousands of projects. It can suggest code for natural language prompts from context and style conventions and can autocomplete functions across dozens of programming languages.
Developers also told us that another major pain point is around catching coding errors. We’ve added a JSON syntax validator that helps catch any errors in the styling and formatting of code. It allows developers to identify and fix errors more quickly, freeing up their time for solving more complex problems.
Convenience is the key
We understand that habits are hard to break and that even the best tools will only be used if they are easily accessible. That’s why we’ve developed a Visual Studio Code extension that allows you to search for code by highlighting the term. We also built a multi-search Chrome extension that allows you to search across four different search engines with a click.
YouCode was created by passionate You.com developers and entrepreneurs to help every developer be the best they can be. Through this process, we have learned a lot.
Building a product that we love was rewarding. But creating the future of search with the people who will benefit most from it — users — is even more satisfying.
Yay, you’ve read the whole article. If you enjoyed this story, please recommend 👏 and share to help others find it!