Data Systems
Search Engines and Elasticsearch
Search is one of the most complex problems in system design. Master inverted indices, relevance scoring, analyzers, and the architecture behind Elasticsearch and Lucene.
- Inverted Index β The data structure that makes full-text search possible
- Relevance β BM25, TF-IDF, and custom scoring functions
- Scalability β Sharding, replication, and distributed search
Search is not just findingβit's finding the most relevant result, fast.
How Search Engines Work
The Inverted Index
Relevance Scoring
BM25 (Elasticsearch Default)
Elasticsearch Architecture
Elasticsearch Key Concepts
| Concept | Description |
|---|---|
| Index | Collection of documents (like a database table) |
| Shard | Partition of an index (horizontal scaling) |
| Replica | Copy of a shard (redundancy + read scaling) |
| Document | JSON object stored in an index |
| Mapping | Schema definition for an index |
| Analyzer | Tokenizes and normalizes text |
Analyzers and Tokenization
| Component | Purpose | Example |
|---|---|---|
| Character Filter | Pre-process text | Strip HTML tags |
| Tokenizer | Split text into terms | Standard tokenizer: "System Design" β ["System", "Design"] |
| Token Filter | Modify terms | Lowercase, stopword removal, stemming |
Search at Scale
Sharding Strategy
Query Execution
- Scatter: Coordinating node sends query to all relevant shards
- Gather: Each shard executes query locally, returns top results
- Merge: Coordinating node merges results from all shards
- Return: Final ranked results returned to client
Practice Exercises
-
Index Design: Design the Elasticsearch mapping for a product catalog with name, description, category, price, and tags. What analyzers would you use for each field?
-
Relevance Tuning: Given a search query "python programming book", how would you configure boosting to prioritize title matches over description matches?
-
Sharding Strategy: You have 500GB of log data to index in Elasticsearch. How many shards and replicas would you create? Justify your numbers.
-
Architecture Design: Design a search-as-you-type feature like Google autocomplete. What are the key components and how do you handle 10K QPS?
What to Learn Next
-> NoSQL Deep Dive Document, key-value, column-family, and graph databases.
-> Kafka Deep Dive Event streaming, partitioning, and exactly-once semantics.
-> Redis Deep Dive Redis data structures, persistence, clustering, and use cases.
-> Search Autocomplete Design Building real-time search autocomplete systems.
-> Search Engine Design Designing a complete search engine like Google.
-> Caching Strategies Cache-aside, write-through, write-back, and cache invalidation.