🎉 75% of content is free forever — Unlock Premium from $10/mo →
CW
💼 Servicesℹ️ About✉️ ContactView Pricing Plansfrom $10

Temporal Queries & SCD — Time Travel, Gap Filling, Sessionization

Advanced SQLTime Series⭐ Premium

Advertisement

Temporal Queries & SCD

Advanced SQL

Master Time in SQL

Temporal queries let you travel through time — answering "what was the state at point X?" and "how did values change?" SCD (Slowly Changing Dimensions) tracks historical changes. Together, they enable time-travel analysis, gap filling, sessionization, and point-in-time joins.

  • SCD Type 1 — Overwrite (no history)
  • SCD Type 2 — Preserve full history with effective/expiry dates
  • Gap Filling — Fill missing dates in time series
  • Sessionization — Group events into user sessions

Temporal data is the backbone of analytics — every fact table has a timestamp, and every dimension needs history.


What Are Temporal Queries?

Temporal queries operate on time-varying data, answering questions about the state of data at specific points in time or across time intervals.

Core Temporal Concepts

  • Bitemporal — Tracks both valid time (when it happened) and transaction time (when it was recorded)
  • SCD Type 2 — Creates new rows for each change, preserving history
  • Gap filling — Ensures continuous time series even when data is missing

SCD Types Visualized

SCD Type 1 (Overwrite)No history preservedJan: John, BostonFeb: John, NYC ← Overwritten!Mar: John, SF ← Overwritten!Result: Only latest value"John lives in SF"No Boston or NYC historySCD Type 2 (Full History)Every change creates a new rowJan 1 – Jan 31John, BostonFeb 1 – Feb 28John, NYCMar 1 – 9999John, SF (current)Query for "What was John's city on Jan 15?"WHERE customer_id = 101AND '2024-01-15' BETWEEN effective_date AND expiry_dateResult: Boston ✓Full history preserved — can answer any point-in-time question

SCD Type 2 Implementation

-- Create SCD Type 2 dimension table
CREATE TABLE dim_customer (
  customer_key SERIAL PRIMARY KEY,
  customer_id INT,
  customer_name VARCHAR(255),
  email VARCHAR(255),
  city VARCHAR(100),
  effective_date DATE,
  expiry_date DATE,
  is_current BOOLEAN DEFAULT true
);

-- Insert new version on change (2-step process)
-- Step 1: Expire current record
UPDATE dim_customer
SET expiry_date = CURRENT_DATE - INTERVAL '1 day',
    is_current = false
WHERE customer_id = 101 AND is_current = true;

-- Step 2: Insert new version
INSERT INTO dim_customer (
  customer_id, customer_name, email, city,
  effective_date, expiry_date, is_current
)
VALUES (
  101, 'John Updated', 'john.new@email.com', 'New York',
  CURRENT_DATE, '9999-12-31', true
);

Point-in-Time Query

-- Find active record for a specific date
SELECT customer_id, customer_name, city
FROM dim_customer
WHERE customer_id = 101
  AND '2024-06-15'::DATE >= effective_date
  AND '2024-06-15'::DATE < expiry_date;

BigQuery Temporal Tables

-- BigQuery SYSTEM_TIME temporal table
CREATE TABLE `project.dataset.orders` (
  order_id INT64, customer_id INT64,
  total_amount NUMERIC, status STRING,
  valid_from TIMESTAMP, valid_to TIMESTAMP
);

-- Query as of specific timestamp
SELECT *
FROM `project.dataset.orders`
FOR SYSTEM_TIME AS OF '2024-06-15 10:00:00'
WHERE order_id = 12345;

-- Query between two timestamps
SELECT *
FROM `project.dataset.orders`
FOR SYSTEM_TIME BETWEEN '2024-01-01' AND '2024-06-30'
WHERE customer_id = 101;

Gap Filling for Time Series

-- Fill missing dates in time series (PostgreSQL)
WITH date_range AS (
  SELECT generate_series(
    '2024-01-01'::DATE,
    '2024-12-31'::DATE,
    '1 day'::INTERVAL
  )::DATE AS sale_date
)
SELECT
  dr.sale_date,
  COALESCE(daily.total_sales, 0) AS total_sales,
  COALESCE(daily.order_count, 0) AS order_count
FROM date_range dr
LEFT JOIN (
  SELECT sale_date, SUM(amount) AS total_sales, COUNT(*) AS order_count
  FROM sales
  WHERE sale_date >= '2024-01-01' AND sale_date <= '2024-12-31'
  GROUP BY sale_date
) daily ON dr.sale_date = daily.sale_date
ORDER BY dr.sale_date;

Sessionization

-- Group events into sessions (30-min inactivity = new session)
WITH events_with_gap AS (
  SELECT
    user_id, event_time, event_type,
    LAG(event_time) OVER (PARTITION BY user_id ORDER BY event_time) AS prev_event_time
  FROM events
),
session_assignments AS (
  SELECT *,
    SUM(CASE
      WHEN event_time - prev_event_time > INTERVAL '30 minutes'
        OR prev_event_time IS NULL
      THEN 1 ELSE 0
    END) OVER (PARTITION BY user_id ORDER BY event_time) AS session_id
  FROM events_with_gap
)
SELECT
  user_id, session_id,
  MIN(event_time) AS session_start,
  MAX(event_time) AS session_end,
  MAX(event_time) - MIN(event_time) AS session_duration,
  COUNT(*) AS event_count
FROM session_assignments
GROUP BY user_id, session_id;

Event Sequencing with Gaps

-- Find gaps in event sequences
WITH numbered_events AS (
  SELECT
    event_id, event_type, event_time,
    LAG(event_time) OVER (PARTITION BY event_type ORDER BY event_time) AS prev_event_time
  FROM events
)
SELECT
  event_type,
  prev_event_time AS gap_start,
  event_time AS gap_end,
  event_time - prev_event_time AS gap_duration,
  CASE
    WHEN event_time - prev_event_time > INTERVAL '1 hour' THEN 'LARGE_GAP'
    ELSE 'NORMAL'
  END AS gap_type
FROM numbered_events
WHERE prev_event_time IS NOT NULL
  AND event_time - prev_event_time > INTERVAL '30 minutes';

Temporal Joins (Point-in-Time)

-- Join two temporal tables for point-in-time analysis
SELECT
  o.order_id, o.order_date,
  c.customer_name, c.city,
  p.product_name, p.category
FROM orders o
INNER JOIN dim_customer c
  ON o.customer_id = c.customer_id
  AND o.order_date >= c.effective_date
  AND o.order_date < c.expiry_date
INNER JOIN dim_product p
  ON o.product_id = p.product_id
  AND o.order_date >= p.effective_date
  AND o.order_date < p.expiry_date;

Month-over-Month Growth

-- Calculate month-over-month growth
WITH monthly_sales AS (
  SELECT
    DATE_TRUNC('month', sale_date)::DATE AS month,
    SUM(amount) AS total_sales
  FROM sales
  GROUP BY 1
)
SELECT
  month, total_sales,
  LAG(total_sales) OVER (ORDER BY month) AS prev_month,
  total_sales - LAG(total_sales) OVER (ORDER BY month) AS absolute_growth,
  ROUND(
    (total_sales - LAG(total_sales) OVER (ORDER BY month)) * 100.0 /
    NULLIF(LAG(total_sales) OVER (ORDER BY month), 0), 2
  ) AS growth_pct
FROM monthly_sales
ORDER BY month;

Quiz: Test Your Knowledge


Follow-Up Questions

  1. How do you handle timezone conversions in temporal queries?
  2. What's the difference between SCD Type 1, 2, and 3?
  3. How would you implement a Type 6 SCD (hybrid approach)?
  4. Explain the concept of bitemporal data and how to query it.
  5. How do you efficiently gap-fill time series data with irregular intervals?

Key Takeaways

  • SCD Type 1 overwrites — use when history doesn't matter
  • SCD Type 2 preserves full history — use for audit trails and point-in-time queries
  • Gap filling — LEFT JOIN against a generated date series to fill missing dates
  • Sessionization — Use LAG + cumulative SUM of gap flags to assign session IDs
  • Temporal joins — Join with BETWEEN effective_date AND expiry_date for point-in-time analysis
  • BigQuery supports FOR SYSTEM_TIME AS OF natively
🔒

Premium Content

Temporal Queries & SCD — Time Travel, Gap Filling, Sessionization

You've previewed the first section. Unlock this full lesson and 900+ advanced tutorials with a Premium plan.

🎯End-to-end Projects
💼Interview Prep
📜Certificates
🤝Community Access

Already a member? Log in

Advertisement