LSTM & Transformer Stock Price Prediction

What is Stock Price Prediction?

Stock price prediction uses statistical and machine learning models to forecast future asset prices based on historical market data and derived features. The efficient market hypothesis (EMH) posits that prices fully reflect available information, making prediction impossible at the weak form. However, empirical evidence shows persistent anomalies — momentum effects, mean reversion in volatility, and microstructure patterns — that exploitable models can capture.

The fundamental challenge is non-stationarity: the statistical properties of financial time series change over time. A model trained on bull-market data may fail catastrophically in a crash. This necessitates rolling retraining windows, regime detection, and robust validation. Deep learning approaches (LSTM, Transformer) excel at capturing long-range temporal dependencies that classical ARIMA models miss, but require careful regularization to avoid overfitting on noisy financial data.

Feature engineering is critical. Raw OHLCV data alone provides limited predictive power. Derived features — relative strength indices, Bollinger Band widths, MACD signals, volume profile, and inter-asset correlations — create a richer representation. The model's job is learning which features matter at which market regimes, not memorizing price patterns.

Project Architecture

Tools & Setup

Step 1: Environment Setup

pip install torch yfinance pandas-ta scikit-learn numpy matplotlib

Step 2: Feature Engineering

Mathematical Foundation

LSTM Cell Update:

Where:

• — forget gate (what to discard from cell state)
• — input gate (what new information to store)
• — output gate (what to output from cell state)
• — sigmoid activation (outputs 0–1)
• — element-wise multiplication

Transformer Attention:

Where:

• — query, key, value matrices
• — key dimension (scaling prevents large dot products)

Model Architecture

Training Pipeline

Performance Results

Real-World Case Study

Two Sigma Investments manages $60B+ using quantitative models. Their approach combines LSTM networks for regime detection with gradient-boosted trees for alpha generation. Public filings show they process 10,000+ features per security, including alternative data (satellite imagery, credit card transactions). Their models retrain weekly on rolling 5-year windows. A simplified version of their approach — LSTM + technical features on S&P 500 constituents — achieves roughly 1.0–1.5 Sharpe, demonstrating the feasibility of deep learning for alpha generation even at retail scale.

Deployment

Common Pitfalls

1. Stationarity violations: Non-stationary price data causes distribution shift — always normalize/standardize per window
2. Look-ahead bias: Target variable leakage when using future data in feature computation
3. Overfitting to noise: Financial data has low signal-to-noise ratio — use aggressive regularization
4. Ignoring transaction costs: A model with 54% accuracy loses money after commissions and slippage
5. Survivorship bias: Training on current index constituents ignores delisted stocks with poor performance

Summary with Key Takeaways

This project built an ensemble LSTM + Transformer stock predictor achieving 58.1% directional accuracy and 1.15 Sharpe ratio. The Transformer's attention mechanism captures longer-range dependencies than LSTM alone, while the ensemble provides robustness. Key insights: technical indicators significantly boost performance over raw prices; Huber loss is more robust to outliers than MSE; and early stopping with 15-epoch patience prevents overfitting on noisy financial data.