LLM Research Paper Guide

The LLM field advances rapidly, with new papers published daily. This guide helps you navigate the literature, understand key contributions, and stay current with the field.

Key Papers by Category

Foundational Papers

Architecture Papers

Key papers:

1. Transformer (Vaswani et al., 2017): Self-attention mechanism
2. GPT (Radford et al., 2018): Decoder-only architecture
3. BERT (Devlin et al., 2018): Encoder-only architecture
4. T5 (Raffel et al., 2019): Encoder-decoder architecture
5. LLaMA (Touvron et al., 2023): Efficient open-source architecture

Training and Alignment

Key papers:

1. InstructGPT (Ouyang et al., 2022): RLHF for instruction following
2. Constitutional AI (Bai et al., 2022): AI-assisted alignment
3. DPO (Rafailov et al., 2023): Direct preference optimization
4. RLHF (Stiennon et al., 2020): Learning from human feedback
5. KTO (Ethayarajh et al., 2024): Kahneman-Tversky optimization

Scaling Laws and Emergence

Key papers:

1. Scaling Laws for Neural LMs (Kaplan et al., 2020): Power law relationships
2. Chinchilla (Hoffmann et al., 2022): Optimal scaling
3. Emergent Abilities (Wei et al., 2022): Abilities that appear at scale
4. Scaling Data-Constrained Language Models (Muennighoff et al., 2023): Data limits

Efficiency and Optimization

Key papers:

1. LoRA (Hu et al., 2022): Low-rank adaptation
2. QLoRA (Dettmers et al., 2023): Quantized LoRA
3. Flash Attention (Dao et al., 2022): Efficient attention
4. GQA (Ainslie et al., 2023): Grouped-query attention
5. Speculative Decoding (Leviathan et al., 2023): Fast inference

Safety and Ethics

Key papers:

1. Training language models to follow instructions (Ouyang et al., 2022)
2. Red Teaming Language Models (Perez et al., 2022)
3. Sleeper Agents (Hubinger et al., 2024): Deceptive alignment
4. Constitutional AI (Bai et al., 2022): Principles for AI behavior

Reading Guide

How to Read a Paper

Reading steps:

1. Skim: Read title, abstract, introduction, conclusion (5-10 minutes)
2. Understand structure: Identify sections and flow
3. Deep read: Read methodology and results carefully
4. Critical analysis: Evaluate assumptions, limitations, reproducibility
5. Synthesis: Connect to other work and your own research

Paper Structure

Critical Reading Questions

Reading Log Template

## Paper Reading Log

### Paper Information
- Title:
- Authors:
- Year:
- Venue:
- Link:

### Summary
- Problem:
- Approach:
- Key contribution:
- Results:

### Key Insights
- Insight 1:
- Insight 2:
- Insight 3:

### Questions
- Question 1:
- Question 2:

### Connection to My Work
- How does this relate to my research?
- What can I apply?

### Rating
- Importance: 1-5
- Quality: 1-5
- Relevance: 1-5

Research Methodology

Research Process

Research phases:

1. Problem identification: Find important open problems
2. Literature review: Understand existing approaches
3. Hypothesis formation: Develop testable hypotheses
4. Experimental design: Plan experiments carefully
5. Implementation: Build and test systems
6. Analysis: Analyze results rigorously
7. Communication: Write and present findings

Experimental Design

Key principles:

1. Baselines: Compare against strong baselines
2. Ablation studies: Understand component contributions
3. Statistical significance: Use proper statistical tests
4. Reproducibility: Provide code, data, and details
5. Multiple runs: Report variance across runs

Evaluation Methodology

Evaluation components:

1. Benchmarks: Standardized evaluation datasets
2. Metrics: Quantitative measures of performance
3. Human evaluation: Subjective quality assessment
4. Safety evaluation: Testing for harmful behaviors
5. Efficiency evaluation: Computational requirements

Common Evaluation Frameworks

Staying Current

Conferences and Venues

Pre-print Servers

1. arXiv: Primary source for ML/NLP papers
2. Semantic Scholar: Search and discovery
3. Papers With Code: Papers with implementations
4. Hugging Face Papers: Curated ML papers

Reading Groups and Communities

Ways to stay current:

1. Daily arXiv scanning: Check new papers daily
2. Reading groups: Join or start a reading group
3. Twitter/X: Follow researchers and discussions
4. Newsletters: Subscribe to ML newsletters
5. Conferences: Attend talks and workshops

Paper Recommendation Systems

Writing Research Papers

Paper Structure

Writing Tips

1. Clear contribution: State your contribution early
2. Motivation: Explain why the problem matters
3. Reproducibility: Provide sufficient details
4. Honest evaluation: Report both strengths and limitations
5. Related work: Properly credit prior work

Common Mistakes

Practical Implementation

Building a Reading System

import requests
import json
from datetime import datetime, timedelta

class PaperReader:
    def __init__(self, api_key: str):
        self.api_key = api_key
        self.base_url = "https://api.semanticscholar.org/graph/v1"
    
    def search_papers(self, query: str, limit: int = 10):
        response = requests.get(
            f"{self.base_url}/paper/search",
            params={
                "query": query,
                "limit": limit,
                "fields": "title,abstract,year,citationCount,url"
            }
        )
        return response.json()["data"]
    
    def get_recent_papers(self, topic: str, days: int = 7):
        cutoff = datetime.now() - timedelta(days=days)
        papers = self.search_papers(topic, limit=50)
        
        recent = []
        for paper in papers:
            if paper.get("year") and paper["year"] >= cutoff.year:
                recent.append(paper)
        
        return recent
    
    def get_citation_network(self, paper_id: str):
        response = requests.get(
            f"{self.base_url}/paper/{paper_id}/citations",
            params={"fields": "title,year"}
        )
        return response.json()["data"]
    
    def create_reading_list(self, papers: list, priority_fn=None):
        if priority_fn:
            papers.sort(key=priority_fn, reverse=True)
        return papers[:10]  # Top 10 papers

Paper Analysis Template

@dataclass
class PaperAnalysis:
    title: str
    authors: list
    year: int
    venue: str
    
    # Summary
    problem: str
    approach: str
    contribution: str
    results: str
    
    # Critical analysis
    strengths: list
    weaknesses: list
    limitations: list
    
    # Personal notes
    key_insights: list
    questions: list
    connections: list
    
    # Rating
    importance: int  # 1-5
    quality: int  # 1-5
    relevance: int  # 1-5

def analyze_paper(paper_path: str) -> PaperAnalysis:
    # Read paper
    with open(paper_path, "r") as f:
        content = f.read()
    
    # Analyze (using LLM or manual analysis)
    analysis = PaperAnalysis(
        title="...",
        authors=[],
        year=2024,
        venue="...",
        problem="...",
        approach="...",
        contribution="...",
        results="...",
        strengths=[],
        weaknesses=[],
        limitations=[],
        key_insights=[],
        questions=[],
        connections=[],
        importance=4,
        quality=4,
        relevance=4
    )
    
    return analysis

Research Log

class ResearchLog:
    def __init__(self):
        self.papers_read = []
        self.ideas = []
        self.experiments = []
        self.writing = []
    
    def add_paper(self, paper_analysis: PaperAnalysis):
        self.papers_read.append({
            "date": datetime.now(),
            "analysis": paper_analysis
        })
    
    def add_idea(self, idea: str, source: str):
        self.ideas.append({
            "date": datetime.now(),
            "idea": idea,
            "source": source
        })
    
    def add_experiment(self, description: str, results: dict):
        self.experiments.append({
            "date": datetime.now(),
            "description": description,
            "results": results
        })
    
    def generate_summary(self):
        return {
            "papers_read": len(self.papers_read),
            "ideas_generated": len(self.ideas),
            "experiments_conducted": len(self.experiments),
            "top_topics": self.get_top_topics()
        }

Best Practices

Reading Practice

1. Consistency: Read regularly, even if just one paper per week
2. Active reading: Take notes and ask questions
3. Discussion: Discuss papers with others
4. Implementation: Try to reimplement key ideas
5. Connection: Connect papers to your own work

Research Practice

1. Rigorous evaluation: Use proper baselines and metrics
2. Reproducibility: Provide code and detailed methods
3. Honesty: Report limitations and negative results
4. Collaboration: Work with others when possible
5. Communication: Write clearly and present well

Practice Exercises

1. Paper Reading: Read a foundational LLM paper (e.g., "Attention Is All You Need") and write a summary with critical analysis.

2. Literature Review: Conduct a literature review on a specific LLM topic. What are the key papers and open problems?

3. Experimental Design: Design an experiment to compare two LLM approaches. What metrics, baselines, and statistical tests would you use?

4. Research Proposal: Write a brief research proposal for an LLM project. What problem would you solve and how?

What to Learn Next