🎉 75% of content is free forever — Unlock Premium from $10/mo →
CW
đŸ’ŧ Servicesâ„šī¸ Aboutâœ‰ī¸ ContactView Pricing Plansfrom $10

Data Serving Patterns: APIs, Materialized Views, Caching, and OLAP

Module 3: Data Warehouses & StorageData ServingđŸŸĸ Free Lesson

Advertisement

Data Serving Patterns: APIs, Materialized Views, Caching, and OLAP

Data serving is the final layer of the data pipeline — it delivers processed data to consumers in the format and latency they need. A data engineer might build the most sophisticated pipeline in the world, but if the data isn't served in a usable format, the pipeline has failed.

Why Data Serving Matters

Data serving solves three problems:

  1. Latency — BI dashboards need sub-second queries; ML training needs bulk reads
  2. Format — Applications need APIs; analysts need SQL; ML needs feature vectors
  3. Cost — Serving 1000 dashboard users is expensive if every query hits the warehouse

The serving layer sits between the data platform and the consumers, optimizing for each consumer's needs.


đŸŽ¯

Interview Question: "How do you reduce latency for BI dashboards?" Answer: (1) Use materialized views for pre-computed aggregations, (2) Implement Redis caching for frequently accessed queries, (3) Use query result caching in the warehouse, (4) Partition tables by date for faster scans, (5) Use columnar storage formats like Parquet.

Serving Pattern Taxonomy

PatternLatencyThroughputConsumerCost
Direct QuerySeconds-MinutesLowAd-hoc analystsLow
Materialized ViewMillisecondsHighBI dashboardsMedium
API ServingMillisecondsHighApplicationsMedium-High
OLAP CubeMillisecondsVery HighInteractive analyticsHigh
Cache LayerMicrosecondsVery HighAll consumersMedium
Feature StoreMillisecondsHighML pipelinesMedium
Stream ServingMillisecondsHighReal-time appsHigh

1. Direct Query (Warehouse as Serving Layer)

The simplest pattern: consumers query the warehouse directly.

ConsumerSQLData WarehouseResultsDashboard

Pros:

  • No additional infrastructure
  • Always up-to-date
  • Single source of truth

Cons:

  • Performance degrades with concurrent users
  • Cost per query is high
  • No caching

When to use: Ad-hoc analysis, small teams, prototyping


2. Materialized Views

A materialized view is a pre-computed result set stored as a physical table. It's the most common serving pattern for BI dashboards.

-- Materialized view for daily sales dashboard
CREATE MATERIALIZED VIEW mv_daily_sales AS
SELECT 
    d.date,
    d.year,
    d.month,
    p.category,
    p.brand,
    c.segment,
    c.country,
    SUM(f.total_amount) as revenue,
    SUM(f.quantity) as units,
    COUNT(DISTINCT f.customer_key) as unique_customers,
    AVG(f.total_amount) as avg_order_value
FROM fact_sales f
JOIN dim_date d ON f.date_key = d.date_key
JOIN dim_product p ON f.product_key = p.product_key
JOIN dim_customer c ON f.customer_key = c.customer_key
GROUP BY 1, 2, 3, 4, 5, 6, 7;

-- Refresh strategy: nightly
REFRESH MATERIALIZED VIEW mv_daily_sales;

Refresh Strategies:

StrategyLatencyUse Case
Full RefreshHoursSmall views, nightly
IncrementalMinutesLarge views, hourly
On-DemandReal-timeLogical views
ConcurrentReal-timeProduction dashboards

3. API Serving (REST/GraphQL)

Expose data through REST or GraphQL APIs for application consumption.

# FastAPI data serving endpoint
from fastapi import FastAPI, HTTPException
from pydantic import BaseModel
from datetime import date
import psycopg2

app = FastAPI(title="Data Serving API")

class SalesSummary(BaseModel):
    date: date
    category: str
    revenue: float
    units: int

@app.get("/api/v1/sales/daily", response_model=list[SalesSummary])
async def get_daily_sales(start_date: date, end_date: date, category: str = None):
    conn = psycopg2.connect("dbname=sales_mart host=warehouse-cluster")
    cur = conn.cursor()
    
    query = """
        SELECT date, category, revenue, units
        FROM mv_daily_sales
        WHERE date BETWEEN %s AND %s
    """
    params = [start_date, end_date]
    
    if category:
        query += " AND category = %s"
        params.append(category)
    
    cur.execute(query, params)
    results = [SalesSummary(*row) for row in cur.fetchall()]
    conn.close()
    return results

@app.get("/api/v1/sales/top-products")
async def get_top_products(limit: int = 10):
    conn = psycopg2.connect("dbname=sales_mart host=warehouse-cluster")
    cur = conn.cursor()
    cur.execute("""
        SELECT p.product_name, SUM(f.total_amount) as revenue
        FROM fact_sales f
        JOIN dim_product p ON f.product_key = p.product_key
        GROUP BY 1
        ORDER BY 2 DESC
        LIMIT %s
    """, (limit,))
    results = [{"product": row[0], "revenue": row[1]} for row in cur.fetchall()]
    conn.close()
    return results

📝

Production Tip: This code example demonstrates core concepts. In production, add error handling, logging, and monitoring. Always validate inputs and handle edge cases.

4. OLAP Cube Serving

OLAP (Online Analytical Processing) cubes pre-aggregate data across multiple dimensions for instant drill-down queries.

OLAP Cube(Pre-aggregated)TimeProductRegion

OLAP Operations:

  • Slice: Select a single dimension (e.g., Q1 2024)
  • Dice: Select a sub-cube (e.g., Q1 2024, Electronics, North America)
  • Drill-down: Move from summary to detail (e.g., Year → Quarter → Month)
  • Roll-up: Move from detail to summary (e.g., Month → Quarter → Year)
  • Pivot: Rotate the cube to view different dimensions

Tools: Apache Druid, ClickHouse, Apache Pinot, StarRocks


5. Cache Layer (Redis/ElastiCache)

A cache layer sits between consumers and the data platform, storing frequently accessed data in memory.

ConsumerCacheHit?YesReturn CachedNoQuery WarehouseCache ResultReturn Data
# Redis caching pattern for data serving
import redis
import json
from datetime import timedelta

redis_client = redis.Redis(host='cache-cluster', port=6379, db=0)

def get_sales_summary(date: str, cache_ttl: int = 3600):
    cache_key = f"sales_summary:{date}"
    
    # Check cache first
    cached = redis_client.get(cache_key)
    if cached:
        return json.loads(cached)
    
    # Cache miss: query warehouse
    conn = psycopg2.connect("dbname=sales_mart host=warehouse-cluster")
    cur = conn.cursor()
    cur.execute("""
        SELECT category, SUM(revenue), SUM(units)
        FROM mv_daily_sales
        WHERE date = %s
        GROUP BY 1
    """, (date,))
    
    results = [{"category": row[0], "revenue": row[1], "units": row[2]} 
               for row in cur.fetchall()]
    conn.close()
    
    # Cache the result
    redis_client.setex(cache_key, timedelta(seconds=cache_ttl), json.dumps(results))
    
    return results

Cache Strategies:

StrategyDescriptionUse Case
Cache-AsideApp checks cache, falls back to DBGeneral purpose
Write-ThroughWrite to cache and DB simultaneouslyHigh consistency
Write-BehindWrite to cache, async write to DBHigh write throughput
Read-ThroughCache fetches from DB on missSimple read patterns

6. Feature Serving (ML)

Feature stores serve pre-computed features to ML training and inference pipelines.

# Feature store pattern using AWS SageMaker Feature Store
import boto3

featurestore_client = boto3.client('sagemaker-featurestore-runtime')

def get_customer_features(customer_id: str):
    """Retrieve pre-computed features for ML inference"""
    response = featurestore_client.get_record(
        FeatureGroupName='customer_features',
        RecordIdentifierValueAsString=customer_id
    )
    
    features = {}
    for feature in response['Record']:
        features[feature['FeatureName']] = feature['ValueAsString']
    
    return features

# Features served: customer_lifetime_value, avg_order_value, 
# purchase_frequency, days_since_last_purchase, segment_score

Feature Store Patterns:

PatternLatencyUse Case
Online StoreMillisecondsReal-time inference
Offline StoreSeconds-MinutesBatch training
Point-in-TimeMinutesHistorical training

7. Stream Serving (Real-Time)

For real-time dashboards and applications, serve data through streaming platforms.

Data StreamProcessMaterializeDashboardKafka/KinesisRedis/DynamoDB
# Real-time aggregation with Kinesis Data Analytics
CREATE OR REPLACE STREAM sales_aggregated_stream (
    category VARCHAR(100),
    window_start TIMESTAMP,
    window_end TIMESTAMP,
    total_revenue DECIMAL(12,2),
    order_count INTEGER
);

CREATE OR REPLACE PUMP sales_aggregated_pump AS
INSERT INTO sales_aggregated_stream
SELECT 
    category,
    TUMBLE_START(proctime, INTERVAL '5' MINUTE) as window_start,
    TUMBLE_END(proctime, INTERVAL '5' MINUTE) as window_end,
    SUM(total_amount) as total_revenue,
    COUNT(*) as order_count
FROM sales_stream
GROUP BY category, TUMBLE(proctime, INTERVAL '5' MINUTE);

Serving Pattern Selection Guide

ConsumerRecommended PatternLatency Target
BI DashboardMaterialized View + Cache< 1 second
REST APIAPI + Cache< 100ms
ML TrainingFeature Store (Offline)Minutes
ML InferenceFeature Store (Online)< 50ms
Real-time DashboardStream Serving< 5 seconds
Ad-hoc AnalysisDirect Query< 30 seconds
Executive ReportsPre-aggregated Views< 5 seconds
Mobile AppAPI + Cache< 200ms

Best Practices

  1. Match serving pattern to consumer — Don't serve BI dashboards through APIs
  2. Implement caching — Cache frequently accessed data to reduce warehouse load
  3. Use materialized views — Pre-compute common aggregations for dashboards
  4. Monitor serving latency — Track P50, P95, P99 latency per serving endpoint
  5. Implement TTL — Set appropriate cache TTL based on data freshness requirements
  6. Use connection pooling — Don't create new connections per request
  7. Implement circuit breakers — Protect downstream services from cascading failures
  8. Version your APIs — Use versioned endpoints for backward compatibility
  9. Document serving contracts — Define SLAs per serving endpoint
  10. Test serving under load — Stress test serving endpoints before production

Knowledge Check

Key Takeaways

  • Data serving is the final layer connecting data platforms to consumers
  • Different consumers need different patterns (APIs, views, caches, streams)
  • Materialized views are the most common pattern for BI dashboards
  • Caching reduces warehouse load and improves latency
  • Feature stores serve ML pipelines with point-in-time correct features
  • Match serving pattern to consumer latency and format requirements

See Also

Need Expert Data Engineering Help?

Get personalized tutoring, project support, or professional consulting.

Advertisement