Data Serving Patterns: APIs, Materialized Views, Caching, and OLAP
Data serving is the final layer of the data pipeline â it delivers processed data to consumers in the format and latency they need. A data engineer might build the most sophisticated pipeline in the world, but if the data isn't served in a usable format, the pipeline has failed.
Why Data Serving Matters
Data serving solves three problems:
- Latency â BI dashboards need sub-second queries; ML training needs bulk reads
- Format â Applications need APIs; analysts need SQL; ML needs feature vectors
- Cost â Serving 1000 dashboard users is expensive if every query hits the warehouse
The serving layer sits between the data platform and the consumers, optimizing for each consumer's needs.
đ¯
Interview Question: "How do you reduce latency for BI dashboards?" Answer: (1) Use materialized views for pre-computed aggregations, (2) Implement Redis caching for frequently accessed queries, (3) Use query result caching in the warehouse, (4) Partition tables by date for faster scans, (5) Use columnar storage formats like Parquet.
Serving Pattern Taxonomy
| Pattern | Latency | Throughput | Consumer | Cost |
|---|---|---|---|---|
| Direct Query | Seconds-Minutes | Low | Ad-hoc analysts | Low |
| Materialized View | Milliseconds | High | BI dashboards | Medium |
| API Serving | Milliseconds | High | Applications | Medium-High |
| OLAP Cube | Milliseconds | Very High | Interactive analytics | High |
| Cache Layer | Microseconds | Very High | All consumers | Medium |
| Feature Store | Milliseconds | High | ML pipelines | Medium |
| Stream Serving | Milliseconds | High | Real-time apps | High |
1. Direct Query (Warehouse as Serving Layer)
The simplest pattern: consumers query the warehouse directly.
Pros:
- No additional infrastructure
- Always up-to-date
- Single source of truth
Cons:
- Performance degrades with concurrent users
- Cost per query is high
- No caching
When to use: Ad-hoc analysis, small teams, prototyping
2. Materialized Views
A materialized view is a pre-computed result set stored as a physical table. It's the most common serving pattern for BI dashboards.
-- Materialized view for daily sales dashboard
CREATE MATERIALIZED VIEW mv_daily_sales AS
SELECT
d.date,
d.year,
d.month,
p.category,
p.brand,
c.segment,
c.country,
SUM(f.total_amount) as revenue,
SUM(f.quantity) as units,
COUNT(DISTINCT f.customer_key) as unique_customers,
AVG(f.total_amount) as avg_order_value
FROM fact_sales f
JOIN dim_date d ON f.date_key = d.date_key
JOIN dim_product p ON f.product_key = p.product_key
JOIN dim_customer c ON f.customer_key = c.customer_key
GROUP BY 1, 2, 3, 4, 5, 6, 7;
-- Refresh strategy: nightly
REFRESH MATERIALIZED VIEW mv_daily_sales;
Refresh Strategies:
| Strategy | Latency | Use Case |
|---|---|---|
| Full Refresh | Hours | Small views, nightly |
| Incremental | Minutes | Large views, hourly |
| On-Demand | Real-time | Logical views |
| Concurrent | Real-time | Production dashboards |
3. API Serving (REST/GraphQL)
Expose data through REST or GraphQL APIs for application consumption.
# FastAPI data serving endpoint
from fastapi import FastAPI, HTTPException
from pydantic import BaseModel
from datetime import date
import psycopg2
app = FastAPI(title="Data Serving API")
class SalesSummary(BaseModel):
date: date
category: str
revenue: float
units: int
@app.get("/api/v1/sales/daily", response_model=list[SalesSummary])
async def get_daily_sales(start_date: date, end_date: date, category: str = None):
conn = psycopg2.connect("dbname=sales_mart host=warehouse-cluster")
cur = conn.cursor()
query = """
SELECT date, category, revenue, units
FROM mv_daily_sales
WHERE date BETWEEN %s AND %s
"""
params = [start_date, end_date]
if category:
query += " AND category = %s"
params.append(category)
cur.execute(query, params)
results = [SalesSummary(*row) for row in cur.fetchall()]
conn.close()
return results
@app.get("/api/v1/sales/top-products")
async def get_top_products(limit: int = 10):
conn = psycopg2.connect("dbname=sales_mart host=warehouse-cluster")
cur = conn.cursor()
cur.execute("""
SELECT p.product_name, SUM(f.total_amount) as revenue
FROM fact_sales f
JOIN dim_product p ON f.product_key = p.product_key
GROUP BY 1
ORDER BY 2 DESC
LIMIT %s
""", (limit,))
results = [{"product": row[0], "revenue": row[1]} for row in cur.fetchall()]
conn.close()
return results
đ
Production Tip: This code example demonstrates core concepts. In production, add error handling, logging, and monitoring. Always validate inputs and handle edge cases.
4. OLAP Cube Serving
OLAP (Online Analytical Processing) cubes pre-aggregate data across multiple dimensions for instant drill-down queries.
OLAP Operations:
- Slice: Select a single dimension (e.g., Q1 2024)
- Dice: Select a sub-cube (e.g., Q1 2024, Electronics, North America)
- Drill-down: Move from summary to detail (e.g., Year â Quarter â Month)
- Roll-up: Move from detail to summary (e.g., Month â Quarter â Year)
- Pivot: Rotate the cube to view different dimensions
Tools: Apache Druid, ClickHouse, Apache Pinot, StarRocks
5. Cache Layer (Redis/ElastiCache)
A cache layer sits between consumers and the data platform, storing frequently accessed data in memory.
# Redis caching pattern for data serving
import redis
import json
from datetime import timedelta
redis_client = redis.Redis(host='cache-cluster', port=6379, db=0)
def get_sales_summary(date: str, cache_ttl: int = 3600):
cache_key = f"sales_summary:{date}"
# Check cache first
cached = redis_client.get(cache_key)
if cached:
return json.loads(cached)
# Cache miss: query warehouse
conn = psycopg2.connect("dbname=sales_mart host=warehouse-cluster")
cur = conn.cursor()
cur.execute("""
SELECT category, SUM(revenue), SUM(units)
FROM mv_daily_sales
WHERE date = %s
GROUP BY 1
""", (date,))
results = [{"category": row[0], "revenue": row[1], "units": row[2]}
for row in cur.fetchall()]
conn.close()
# Cache the result
redis_client.setex(cache_key, timedelta(seconds=cache_ttl), json.dumps(results))
return results
Cache Strategies:
| Strategy | Description | Use Case |
|---|---|---|
| Cache-Aside | App checks cache, falls back to DB | General purpose |
| Write-Through | Write to cache and DB simultaneously | High consistency |
| Write-Behind | Write to cache, async write to DB | High write throughput |
| Read-Through | Cache fetches from DB on miss | Simple read patterns |
6. Feature Serving (ML)
Feature stores serve pre-computed features to ML training and inference pipelines.
# Feature store pattern using AWS SageMaker Feature Store
import boto3
featurestore_client = boto3.client('sagemaker-featurestore-runtime')
def get_customer_features(customer_id: str):
"""Retrieve pre-computed features for ML inference"""
response = featurestore_client.get_record(
FeatureGroupName='customer_features',
RecordIdentifierValueAsString=customer_id
)
features = {}
for feature in response['Record']:
features[feature['FeatureName']] = feature['ValueAsString']
return features
# Features served: customer_lifetime_value, avg_order_value,
# purchase_frequency, days_since_last_purchase, segment_score
Feature Store Patterns:
| Pattern | Latency | Use Case |
|---|---|---|
| Online Store | Milliseconds | Real-time inference |
| Offline Store | Seconds-Minutes | Batch training |
| Point-in-Time | Minutes | Historical training |
7. Stream Serving (Real-Time)
For real-time dashboards and applications, serve data through streaming platforms.
# Real-time aggregation with Kinesis Data Analytics
CREATE OR REPLACE STREAM sales_aggregated_stream (
category VARCHAR(100),
window_start TIMESTAMP,
window_end TIMESTAMP,
total_revenue DECIMAL(12,2),
order_count INTEGER
);
CREATE OR REPLACE PUMP sales_aggregated_pump AS
INSERT INTO sales_aggregated_stream
SELECT
category,
TUMBLE_START(proctime, INTERVAL '5' MINUTE) as window_start,
TUMBLE_END(proctime, INTERVAL '5' MINUTE) as window_end,
SUM(total_amount) as total_revenue,
COUNT(*) as order_count
FROM sales_stream
GROUP BY category, TUMBLE(proctime, INTERVAL '5' MINUTE);
Serving Pattern Selection Guide
| Consumer | Recommended Pattern | Latency Target |
|---|---|---|
| BI Dashboard | Materialized View + Cache | < 1 second |
| REST API | API + Cache | < 100ms |
| ML Training | Feature Store (Offline) | Minutes |
| ML Inference | Feature Store (Online) | < 50ms |
| Real-time Dashboard | Stream Serving | < 5 seconds |
| Ad-hoc Analysis | Direct Query | < 30 seconds |
| Executive Reports | Pre-aggregated Views | < 5 seconds |
| Mobile App | API + Cache | < 200ms |
Best Practices
- Match serving pattern to consumer â Don't serve BI dashboards through APIs
- Implement caching â Cache frequently accessed data to reduce warehouse load
- Use materialized views â Pre-compute common aggregations for dashboards
- Monitor serving latency â Track P50, P95, P99 latency per serving endpoint
- Implement TTL â Set appropriate cache TTL based on data freshness requirements
- Use connection pooling â Don't create new connections per request
- Implement circuit breakers â Protect downstream services from cascading failures
- Version your APIs â Use versioned endpoints for backward compatibility
- Document serving contracts â Define SLAs per serving endpoint
- Test serving under load â Stress test serving endpoints before production
Knowledge Check
Key Takeaways
- Data serving is the final layer connecting data platforms to consumers
- Different consumers need different patterns (APIs, views, caches, streams)
- Materialized views are the most common pattern for BI dashboards
- Caching reduces warehouse load and improves latency
- Feature stores serve ML pipelines with point-in-time correct features
- Match serving pattern to consumer latency and format requirements