Introduction to Time Series
Time series data consists of observations recorded sequentially over time. Unlike cross-sectional data where observations are independent, time series observations are typically related to their predecessors. This temporal dependency creates both analytical challenges and unique forecasting opportunities.
Time series analysis addresses questions about how systems evolve over time, what patterns characterize the series, and how to predict future values. Applications span finance (stock prices), economics (unemployment rates), climate science (temperature records), and many other domains.
Understanding time series characteristics is essential before modeling. Components including trend, seasonality, and autocorrelation must be identified. This understanding guides appropriate model selection and interpretation.
Components of Time Series
Time series often exhibit multiple components that can be separated for analysis. Understanding these components enables appropriate modeling strategies.
Trend Components
Trend represents long-term movement in the series. It might be increasing, decreasing, or stable. Trends might be deterministic (following a specific pattern) or stochastic (random walk with drift).
Linear trends show constant rates of change. Polynomial trends allow more complex shapes. Piecewise trends allow different patterns in different periods.
Trend identification requires distinguishing true trends from other patterns. Moving averages can reveal trends by smoothing short-term fluctuations. Regression on time provides formal trend estimates.
Seasonal Components
Seasonal patterns repeat at fixed intervals. Annual seasonality repeats yearly, weekly repeats weekly, and so forth. Seasonal patterns are common in economic and environmental data.
Seasonal variation might be additive or multiplicative. Additive seasonality adds constant seasonal effects regardless of level. Multiplicative seasonality scales with the series level.
Seasonal indices quantify seasonal effects. They might be added to or multiplied with the trend to create seasonal patterns. Seasonal decomposition separates these components.
Cyclical Components
Cyclical patterns repeat at irregular intervals, unlike seasonal patterns with fixed periods. Business cycles of expansion and contraction are classic examples.
Cyclical patterns are often harder to model than seasonal patterns because the period is unknown. Spectral analysis can identify cyclical frequencies, though interpretation requires care.
Cycle identification often requires additional information beyond the series itself. Economic indicators might help identify turning points.
Random Components
Random (irregular, residual) components represent unpredictable variation after accounting for other components. They might reflect measurement error, rare events, or inherent system randomness.
White noise represents random variation with no pattern. It has constant variance and zero autocorrelation. Many models assume normally distributed (Gaussian) white noise.
Understanding the nature of random components helps model selection. Models might explicitly model some random components as autoregressive processes.
Stationarity
Stationarity is a fundamental concept in time series analysis. It describes whether statistical properties remain constant over time.
Strict Stationarity
Strict stationarity requires that the joint distribution of any set of time points depends only on their relative positions, not absolute times. All statistical properties remain constant over time.
This is a strong condition rarely satisfied in practice. Most time series are not strictly stationary. However, the concept provides a useful idealization.
Weak Stationarity
Weak stationarity (also called second-order stationarity) requires constant mean, constant variance, and autocovariance that depends only on lag, not on absolute time.
Many time series models assume weak stationarity. Non-stationary series require transformation to achieve stationarity before modeling.
Checking stationarity involves examining time-varying mean and variance. Visual plots and statistical tests (like Augmented Dickey-Fuller) help assess stationarity.
Achieving Stationarity
Differencing removes trends by computing successive differences. First differencing removes linear trends. Second differencing might be needed for more complex trends.
Detrending involves fitting a trend model and subtracting the fitted values. This removes deterministic trends while preserving other components.
Log transformation can stabilize variance when variance increases with the level. This is common in financial data.
Autocorrelation
Autocorrelation measures correlation between observations at different lags. It is central to time series modeling because it captures temporal dependencies.
Autocorrelation Function (ACF)
The ACF plots autocorrelation against lag. Each lag shows correlation between observations separated by that time period. Significant correlations at many lags indicate temporal dependencies.
The ACF helps identify patterns. Slow decay suggests non-stationarity. Seasonality shows spikes at seasonal lags. Short-term persistence shows decay from high first lag.
The 95% confidence band provides significance thresholds. Correlations within the band could arise by chance. Those outside suggest real dependencies.
Partial Autocorrelation Function (PACF)
The PACF measures correlation at each lag after removing the effect of intermediate lags. It shows the direct relationship at each lag.
The PACF helps identify autoregressive order. Significant PACF values at early lags followed by non-significant values indicate the autoregressive order.
For AR(p) processes, the PACF is significant through lag p then cuts off. The ACF decays gradually. This contrast helps identify model order.
Ljung-Box Test
The Ljung-Box test evaluates whether a group of autocorrelations is significantly different from zero. It tests overall serial correlation rather than individual lags.
The test applies to residuals to check whether modeling has removed autocorrelation. Significant Ljung-Box results indicate remaining pattern.
The test should be applied to model residuals, not raw data. Many packages provide automatic testing.
Autoregressive (AR) Models
AR models express current values as linear combinations of past values plus noise. They capture temporal dependencies through past observations.
AR Model Specification
An AR(p) model expresses the current value as a linear combination of p previous values plus white noise. The model includes p parameters for the lagged values plus an intercept and noise variance.
The order p determines how many past values influence the current value. Selection involves balancing fit against complexity using criteria like AIC or BIC.
AR models assume stationarity. They can represent many stationary time series patterns. Non-stationary series require differencing before AR modeling.
Estimation and Diagnostics
AR model estimation uses least squares or maximum likelihood. Parameter estimates indicate how much each lag contributes to predicting the current value.
Diagnostics check residual autocorrelation. Residuals should be white noise. Significant autocorrelation indicates inadequate model specification.
Forecasting with AR models uses recursive prediction. The forecast for the next period uses estimated parameters and recent values.
Moving Average (MA) Models
MA models express current values as linear combinations of past noise terms. They capture short-term effects through recent innovations.
MA Model Specification
An MA(q) model expresses the current value as a linear combination of q recent white noise terms. The order q determines how many past innovations affect the current value.
The model assumes white noise innovations. Each innovation represents a surprise (unpredictable part) from the previous period.
MA(q) processes are always stationary regardless of parameter values. This is different from AR processes.
ACF and PACF
MA(q) processes show distinctive patterns in ACF and PACF. The ACF cuts off after lag q. The PACF decays gradually.
This contrasts with AR(p) processes: ACF decays gradually while PACF cuts off after lag p. The patterns help identify model type and order.
Combining AR and MA components (ARMA models) can represent various patterns with more efficient parameter use.
ARIMA Models
ARIMA combines autoregressive and moving average components with differencing to handle non-stationarity. It provides a general modeling framework.
Model Components
An ARIMA(p,d,q) model includes p AR terms, d differencing orders, and q MA terms. The differencing makes the series stationary before modeling.
The Box-Jenkins methodology provides a systematic approach: identification (using ACF/PACF), estimation, and diagnostics. This classic approach remains influential.
Parameter estimation uses maximum likelihood. The likelihood is computed treating initial values appropriately. Different approaches yield slightly different estimates.
Model Selection
Information criteria (AIC, BIC) balance fit and complexity. Lower values indicate better models. They penalize additional parameters, preventing overfitting.
Automatic selection procedures search over model orders and select best according to criteria. This is implemented in most time series software.
Diagnostic checking evaluates residuals. ACF of residuals should show no pattern. Ljung-Box test should not reject white noise.
Seasonal Models
Seasonal patterns require specialized models that account for periodic structure. Seasonal ARIMA (SARIMA) extends ARIMA with seasonal components.
SARIMA Model Structure
A SARIMA(p,d,q)(P,D,Q)s model includes non-seasonal AR(p), differencing(d), MA(q) components and seasonal AR(P), differencing (D), MA(Q) components with seasonal period s.
The seasonal period might be 12 for monthly data with annual seasonality, 4 for quarterly data, 7 for daily data with weekly seasonality.
Parameters multiply when interpreting: seasonal AR(1) at lag 12 involves observation from 12 periods ago.
Model Identification
Identifying seasonal models requires examining both seasonal and non-seasonal autocorrelations. The seasonal part appears at lags equal to the seasonal period.
Similar to non-seasonal models, ACF and PACF help identify seasonal orders. The seasonal patterns are scaled versions of non-seasonal patterns.
Fitting and diagnostics proceed similarly to non-seasonal models. Residual checking should examine both seasonal and non-seasonal lags.
Exponential Smoothing
Exponential smoothing methods provide alternative approaches to time series forecasting. They use weighted averages of past observations with exponentially decaying weights.
Simple Exponential Smoothing
Simple exponential smoothing applies to series without trend or seasonality. The forecast is a weighted average of all past observations with exponentially decaying weights.
The smoothing parameter (alpha) controls the rate of decay. Values near 1 give more weight to recent observations. Values near 0 give more weight to distant observations.
The method produces flat forecasts. It does not extrapolate trends, only captures recent level.
Holt's Method
Holt's method extends exponential smoothing to series with trends. It uses separate smoothing parameters for level and trend. This enables trend extrapolation.
The method produces forecasts that continue the trend. Linear trends are typical. Damped trends that gradually flatten are also possible.
Holt's method can accommodate seasonality by adding a third smoothing component. This creates the Holt-Winters method.
State Space Formulation
Exponential smoothing methods can be expressed in state space form. This unified framework connects exponential smoothing to other approaches.
The state space formulation provides standard errors and confidence intervals. It enables likelihood-based model selection. It connects exponential smoothing to ARIMA models.
Forecasting Methods
Forecasting involves predicting future values. Different methods suit different situations, and forecast accuracy depends on method appropriateness.
Point Forecasts
Point forecasts predict single values for future periods. They minimize expected squared error (mean), absolute error (median), or other loss functions.
Different models produce different forecasts. Forecasts should be evaluated in comparison to actual values when possible.
Forecast accuracy typically declines with forecast horizon. Short-term forecasts are more reliable than long-term forecasts.
Prediction Intervals
Prediction intervals quantify forecast uncertainty. They provide ranges likely to contain future values. Wider intervals for longer horizons reflect increasing uncertainty.
Assumptions matter. Assumed error distributions affect interval width. Model misspecification can produce misleading intervals.
Simulation methods like bootstrap can provide intervals without distributional assumptions. They sample from predictive distributions.
Evaluation
Forecast accuracy measures compare predictions to actual values. Mean Absolute Error (MAE), Mean Squared Error (MSE), and Mean Absolute Percentage Error (MAPE) are common measures.
Different measures emphasize different aspects. MSE heavily penalizes large errors. MAPE provides percentage errors useful for comparison across series.
Out-of-sample evaluation prevents overfitting. Holding out recent data for testing provides honest accuracy estimates.
Key Takeaways
- Time series have trend, seasonal, cyclical, and random components that can be separated
- Stationarity (constant mean, variance, autocovariance) is required for many models
- Autocorrelation measures temporal dependencies and helps identify model types
- ARIMA models combine autoregressive, differencing, and moving average components
- Seasonal models extend ARIMA with periodic structure
- Exponential smoothing provides alternative approaches with different assumptions