What Are Large Language Models?

Large Language Models (LLMs) represent a paradigm shift in artificial intelligence—deep neural networks trained on massive text corpora that exhibit remarkable abilities in language understanding, generation, and reasoning. This tutorial provides a comprehensive foundation for understanding what LLMs are, how they scale, and why they matter.

The Scale of LLMs

The defining characteristic of LLMs is scale—not just in parameters, but across three dimensions:

Scale in Practice

Emergent Capabilities

One of the most fascinating properties of LLMs is emergence: abilities that appear suddenly as model scale increases, which are absent in smaller models.

Key emergent capabilities include:

• In-context learning: Learning from examples provided in the prompt without gradient updates
• Chain-of-thought reasoning: Breaking complex problems into intermediate reasoning steps
• Instruction following: Generalizing to unseen instruction formats
• Code generation: Writing and debugging programs from natural language descriptions
• Multilingual transfer: Performing tasks in languages not explicitly represented in training

A Brief History of LLMs

The Pre-Transformer Era

Language modeling has roots in statistical methods (n-gram models) and early neural approaches (RNNs, LSTMs). The key limitation was the inability to capture long-range dependencies effectively.

The Transformer Revolution (2017)

The "Attention Is All You Need" paper (Vaswani et al., 2017) introduced the Transformer architecture, replacing recurrence with self-attention. This enabled:

• Parallelized training on modern hardware
• Better capture of long-range dependencies
• Scalability to billions of parameters

GPT-2 and the Zero-Shot Era (2019)

OpenAI's GPT-2 (1.5B parameters) demonstrated that a language model trained on a large corpus could perform many tasks zero-shot, simply by conditioning on a natural language prompt.

GPT-3 and In-Context Learning (2020)

GPT-3 (175B parameters) introduced the concept of in-context learning—providing a few examples in the prompt enables the model to learn new tasks without fine-tuning. This was a paradigm shift from task-specific fine-tuning to general-purpose prompting.

ChatGPT and the Instruction Following Era (2022)

ChatGPT combined GPT-3.5 with Reinforcement Learning from Human Feedback (RLHF), creating a model that could follow complex instructions and engage in multi-turn dialogue. This was the first LLM to achieve mainstream consumer adoption.

GPT-4 and Multimodal Models (2023)

GPT-4 introduced multimodal capabilities (text + images) and demonstrated significant improvements in reasoning, coding, and factual accuracy.

The Open-Source Revolution (2023-2024)

Models like LLaMA, Mistral, and Qwen demonstrated that open-source models could compete with proprietary ones. This democratized LLM research and enabled rapid innovation.

When to Use LLMs vs Traditional ML

Understanding when LLMs are appropriate is crucial for practical applications:

LLMs Excel At:

• Few-shot or zero-shot tasks: When you have limited labeled data
• Open-ended generation: Creative writing, brainstorming, summarization
• Complex reasoning: Tasks requiring multi-step logical deduction
• Code generation: Writing, debugging, and explaining code
• Natural language interfaces: Chatbots, assistants, search

Traditional ML Is Often Better For:

• Structured prediction: When you have large labeled datasets
• Latency-critical applications: When inference speed is paramount
• Resource-constrained environments: Edge devices, mobile
• Well-defined classification/regression: When a simple model suffices

Mathematical Foundations

The core objective of language modeling is to estimate the probability distribution over tokens in a sequence:

The model is trained by minimizing the negative log-likelihood:

Practical Example: Loading an LLM

Here's a minimal example of loading and using an LLM with HuggingFace Transformers:

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

# Load a pre-trained LLM
model_name = "meta-llama/Llama-2-7b-chat-hf"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype=torch.float16,
    device_map="auto"
)

# Create a prompt
messages = [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "Explain the concept of emergence in LLMs."}
]

# Tokenize and generate
inputs = tokenizer.apply_chat_template(
    messages, 
    return_tensors="pt",
    add_generation_prompt=True
).to(model.device)

outputs = model.generate(
    inputs,
    max_new_tokens=512,
    temperature=0.7,
    top_p=0.9,
    do_sample=True
)

# Decode and print
response = tokenizer.decode(outputs[0][inputs.shape[-1]:], skip_special_tokens=True)
print(response)

Practice Exercises

1. Conceptual: Explain why scaling laws suggest that both model size and data size must increase together for optimal performance. What happens if you scale only one?

2. Mathematical: Given a vocabulary of size V = 50,000 and a sequence of length T = 1024, calculate the total number of parameters in the output projection layer of a model with hidden dimension d = 4096.

3. Practical: Using the HuggingFace Transformers library, load a small GPT-2 model and measure the difference in perplexity between GPT-2 (124M) and GPT-2 Medium (355M) on a text sample of your choice.

4. Research: Compare the training compute requirements of GPT-3 (175B) and LLaMA 2 70B. Which model is more compute-efficient and why?

What to Learn Next