Turning conversations into useful understanding

Overview

Reddit knew where conversations happened, but had limited understanding of what they were actually about.

Search, Discovery, Recommendations, Advertising, and emerging AI products increasingly needed the same semantic capability, but teams were building against different models and systems. I worked across Product, Engineering, Machine Learning, Search, and other organizations to define a shared model for content understanding, turn fragmented semantic capabilities into reusable infrastructure, and use Search to prove its value in production.

Impact

The work turned fragmented semantic capabilities into shared infrastructure and demonstrated that deeper content understanding could create measurable product value. Search became the first production proof point while the underlying system created a foundation for multiple product areas to build upon.

Team & Partners

Search Engineering, Machine Learning, Platform Engineering, Product, Data Science, Advertising, Executive Leadership

Reddit Had Millions of Conversations It Couldn't Fully Understand

Reddit hosts millions of conversations about companies, products, athletes, movies, public figures, locations, technologies, and emerging trends.

Those conversations are one of Reddit's greatest strengths, but much of that knowledge existed only as unstructured text.

Historically, Reddit organized content around where conversations happened through communities.

And those conversations often span hundreds or thousands of communities. Reddit could understand where a conversation lived, but had limited understanding of what that conversation was actually about. That created limitations across the business.

Before Building Technology, We Had to Build Shared Understanding

Different teams were already trying to understand Reddit's content, but they were operating from different mental models.

Topics. Entities. Taxonomy. Interests.

These weren't simply terminology differences. They represented different assumptions about what Reddit's content was and how it should be represented.

I worked across Product, Engineering, Machine Learning, Search, Advertising, and other organizations to establish a shared framework for the different levels of content understanding.

For the first time, teams could reason about semantic understanding using the same primitives and begin designing systems that could work together rather than independently.

We could have started with models and APIs. Instead, we first established what the platform needed to understand and represent.

Platforms don't scale simply because the technology works. They scale when the organization shares the same mental model.

Designing for an Organization That Wasn't Ready for a Platform Yet

Shared understanding gave us a foundation, but long-term platform investment was still uncertain.

Infrastructure competed with highly visible product work, ownership remained fragmented, and we couldn't assume a centralized platform organization would emerge. That constraint shaped the strategy.

Rather than designing an idealized platform that required significant upfront investment, we focused on capabilities that could create immediate value while remaining extensible as adoption grew.

Every decision needed to answer two questions:

• Can this create value today?

• Can other teams build on it tomorrow?

That became our strategy for building the platform incrementally.

Designing the System Behind the System

Modern machine learning made it possible to identify people, products, organizations, places, and other entities across millions of conversations.

But foundational infrastructure couldn't depend entirely on probabilistic systems.

Other teams needed to trust what they were building upon.

So we designed a hybrid architecture combining machine learning, deterministic systems, and human expertise.

Rather than treating machine learning as the source of truth, each layer strengthened the others.

Models could identify and propose semantic relationships at scale. Deterministic systems could preserve consistency. Human expertise could resolve ambiguity and improve strategically important parts of the knowledge system. That mattered because semantic understanding is never finished.

New companies launch. Products evolve. Public figures emerge. Communities invent new language. The platform needed to continuously learn without sacrificing the trust required for other teams to depend on it.

Foundational platforms have to earn trust before they can earn adoption.

Designing Human Expertise Into the Platform

The hybrid architecture created another design problem: How should humans interact with a semantic system operating across millions of conversations?

We designed an internal Entity Manager that made the platform's understanding inspectable and editable.

Curators could review model suggestions, validate entities, resolve ambiguity, and strengthen relationships between people, organizations, products, places, communities, and emerging trends.

The goal wasn't simply to create an administrative database. It was to create a living representation of Reddit's understanding of its own content.

Rather than replacing human expertise with machine learning, we designed the platform so each could continuously improve the other.

Creating Leverage Instead of Ownership

As the platform evolved, a larger organizational question remained: Should content understanding become a centralized platform or remain embedded within individual teams?

Trying to resolve that question upfront would have required significant organizational commitment before we had demonstrated enough value.

So we took a different approach. Rather than asking teams to adopt a platform because we believed it was strategically important, we created reusable capabilities they could choose to build upon.

We didn't need the platform to own every experience. We needed it to make every experience built on top of it more capable. Each successful integration increased both the usefulness of the infrastructure and the evidence for investing further.

But Infrastructure Alone Wasn't Enough

Even with a compelling architecture, infrastructure is difficult to evaluate from the outside.

The organization could understand the theoretical value of semantic understanding. Continued investment required something stronger: Evidence.

I began looking for a product experience where the platform could demonstrate its value directly to users. Search stood out.

At the time, autocomplete primarily helped people navigate to communities. It did much less to help them discover everything else Reddit Search could surface or formulate the query they were actually trying to make.

Typing “Taylor Swift,” for example, could quickly route someone to a community, but it couldn't meaningfully help them continue toward a particular question, event, or idea.

The system understood where content lived. It didn't understand enough about what someone was trying to find. The semantic capabilities we were building could change that.

Turning Search Into Our Proof Point

By this point, we had built a semantic layer capable of identifying and organizing the people, products, organizations, places, and other entities being discussed across Reddit.

Search gave us a way to put that intelligence directly in front of people. NER helped us understand what Reddit's content was about. Query Autocomplete extended that same foundation toward understanding what someone was trying to find. Together, they connected Reddit's semantic understanding of its content with the intent expressed in a search query.

The company hackathon gave us a low-cost way to test that platform thesis before asking for broader investment. One of the first questions was what we should actually optimize for. We weren't trying to predict the single “best” query. That would require continuously defining what “best” meant as language, culture, and Reddit itself changed. Instead, we started with the intent someone had already expressed and helped them continue it.

Working alongside Search engineers, we combined Reddit's existing query stream, SEO traffic, and the semantic understanding we had already built to create richer candidate suggestions. Language models helped normalize spelling, slang, and different formulations of similar intent into a cleaner set of possible continuations.

Safety filtering removed inappropriate or harmful associations, while the resulting suggestions were converted into extremely fast cached lookups. That architecture mattered because autocomplete had an unusually demanding performance constraint. Beyond roughly 300 milliseconds, suggestions could begin to feel delayed or stale.

The prototype showed that we could connect Reddit's understanding of its own content with a person's emerging search intent while still delivering the speed and predictability required by a core product surface.

We worked with Search Engineering to turn the hackathon concept into a production capability capable of operating at Reddit scale.

What began as a hackathon prototype became a production capability improving one of Reddit's most frequently used surfaces. More importantly, it proved something larger: The invisible infrastructure underneath Reddit could create visible, measurable product value.

Outcomes

The outcome was bigger than a better Search experience. We demonstrated that shared semantic infrastructure could create measurable product value, taking semantic query autocomplete from a hackathon prototype into production while also improving time to content and return Search usage.

Underneath the experience, we established reusable content-understanding infrastructure that combined machine learning, deterministic systems, and human expertise. That foundation could support Search, Discovery, Advertising, Reddit Pro, Recommendations, and emerging AI experiences while giving teams a system they could inspect, trust, and extend.

The work also created a shared model for content understanding across previously fragmented teams. Instead of requiring a centralized platform organization upfront, we showed that foundational infrastructure could earn adoption and investment incrementally by making the products built on top of it more capable.

Read more about this project: Query Autocomplete from LLMs, NER you OK?

Key Learnings

• Platforms need shared mental models before shared technology. Once teams shared the same language and primitives, it became much easier to build infrastructure they could collectively depend on.

• Evidence creates momentum. A small product proof created more confidence in semantic infrastructure than a larger strategic argument could have on its own.

• Design for leverage, not ownership. The platform became valuable by making other products more capable rather than centralizing every experience around itself.

More Featured Work