LLM for Information Extraction

Information extraction (IE) is the task of extracting structured information from unstructured text. LLMs have transformed IE by enabling zero-shot and few-shot extraction, cross-domain generalization, and complex schema understanding.

Information Extraction Tasks

Named Entity Recognition (NER)

Standard entity types:

• PER: Person names
• ORG: Organizations
• LOC: Locations
• DATE: Dates and times
• MISC: Miscellaneous entities

Relation Extraction

Event Extraction

Template Filling

Mathematical Formulation

Sequence Labeling for NER

NER uses BIO tagging: B- (beginning), I- (inside), O (outside) for entity boundaries.

Relation Extraction

LLM Approaches to IE

Zero-Shot Extraction

LLMs can extract information without task-specific training by using prompt engineering.

Few-Shot Extraction

Providing a few examples dramatically improves extraction quality.

Structured Output Generation

LLMs can generate structured output formats like JSON, YAML, or XML.

Evaluation Metrics

For NER

For Relation Extraction

Practical Implementation

NER with LLMs

from transformers import AutoTokenizer, AutoModelForCausalLM
import json

model_name = "meta-llama/Llama-3-8B-Instruct"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name, device_map="auto")

text = """Apple Inc. announced today that CEO Tim Cook will visit 
the European headquarters in Dublin, Ireland next week."""

prompt = f"""Extract all named entities from the following text and 
return as JSON with entity type:

Text: {text}

Output:"""

inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=200)
result = tokenizer.decode(outputs[0][inputs.shape[-1]:], skip_special_tokens=True)
print(json.loads(result))

Relation Extraction

def extract_relations(text, entity1, entity2, model, tokenizer):
    prompt = f"""What is the relationship between {entity1} and {entity2} 
in the following text?

Text: {text}

Relationship:"""
    
    inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
    outputs = model.generate(**inputs, max_new_tokens=50)
    return tokenizer.decode(outputs[0][inputs.shape[-1]:], skip_special_tokens=True)

# Example
text = "Elon Musk founded SpaceX in 2002."
relation = extract_relations(text, "Elon Musk", "SpaceX", model, tokenizer)
# Returns: "founded" or "founder_of"

Structured Extraction with Pydantic

from pydantic import BaseModel
from typing import List, Optional
from transformers import AutoTokenizer, AutoModelForCausalLM

class CompanyInfo(BaseModel):
    name: str
    founded_year: Optional[int]
    headquarters: Optional[str]
    ceo: Optional[str]
    industry: Optional[str]

def extract_company_info(text: str, model, tokenizer) -> CompanyInfo:
    prompt = f"""Extract company information from the text and return as JSON:

Text: {text}

CompanyInfo:"""
    
    inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
    outputs = model.generate(**inputs, max_new_tokens=200)
    result = tokenizer.decode(outputs[0][inputs.shape[-1]:], skip_special_tokens=True)
    
    return CompanyInfo.parse_raw(result)

Advanced Techniques

Nested NER

Some entities contain other entities:

Few-Shot with demonstrations

Selecting the right demonstrations significantly impacts performance:

1. Similar examples: Choose demonstrations similar to the target text
2. Diverse examples: Cover different entity types and patterns
3. Balanced examples: Ensure equal representation of entity types

Schema-Guided Extraction

Challenges and Solutions

Ambiguity in Entity Boundaries

Cross-Domain Generalization

LLMs often generalize better across domains than fine-tuned models, but may still struggle with domain-specific terminology.

Hallucination in Extraction

LLMs may extract information not present in the text or misclassify entities.

Best Practices

Prompt Design

1. Clear entity definitions: Define each entity type clearly
2. Output format specification: Specify exact output format
3. Boundary guidelines: Clarify how to handle ambiguous boundaries
4. Error handling: Specify behavior for missing or unclear information

Quality Assurance

1. Human review: Sample and review extractions
2. Consistency checks: Validate extractions against schema
3. Confidence scoring: Track extraction confidence
4. Active learning: Use uncertain cases to improve the system

Practice Exercises

1. NER Evaluation: Compare zero-shot and few-shot NER performance on a benchmark dataset. How many examples are needed for competitive performance?

2. Relation Extraction: Build a relation extraction system for a specific domain (e.g., biomedical, financial). How does domain specificity affect performance?

3. Schema Design: Design a schema for extracting information from news articles. What entity and relation types are needed?

4. Error Analysis: Analyze common extraction errors in your system. What patterns emerge?

What to Learn Next