LLM for Question Answering

Question answering (QA) systems aim to provide accurate answers to natural language questions. LLMs have transformed QA by enabling open-domain question answering, conversational QA, and complex reasoning over multiple information sources.

QA Taxonomy

By Answer Source

By Answer Type

Mathematical Formulation

Extractive QA

The model predicts probability distributions over start and end positions in the context.

Open-Domain QA

The pipeline retrieves relevant documents and generates answers conditioned on them.

Retrieval-Augmented Generation (RAG)

RAG Pipeline

1. Query Processing: Reformulate the question for retrieval
2. Document Retrieval: Find relevant documents from knowledge base
3. Context Integration: Combine retrieved documents with the question
4. Answer Generation: Generate an answer using the LLM
5. Answer Verification: Validate the answer against source documents

Conversational QA

Challenges in Conversational QA

1. Coreference Resolution: "Who is he?" → resolving to a previously mentioned entity
2. Ellipsis: "What about in 2020?" → implicit reference to previous topic
3. Topic Shift: Questions that change the topic mid-conversation
4. Context Maintenance: Tracking relevant information across turns

Evaluation Metrics

Exact Match (EM)

F1 Score

Evaluation Comparison

Practical Implementation

Basic QA with LLMs

from transformers import AutoTokenizer, AutoModelForCausalLM

model_name = "meta-llama/Llama-3-8B-Instruct"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name, device_map="auto")

context = """The Eiffel Tower is a wrought-iron lattice tower in Paris, France.
It was built in 1889 as the centerpiece of the 1889 World's Fair."""

question = "When was the Eiffel Tower built?"

prompt = f"""Context: {context}

Question: {question}

Answer:"""

inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=50)
answer = tokenizer.decode(outputs[0][inputs.shape[-1]:], skip_special_tokens=True)
print(answer.strip())  # "1889"

RAG Implementation with LangChain

from langchain_community.vectorstores import Chroma
from langchain_community.embeddings import HuggingFaceEmbeddings
from langchain_community.llms import HuggingFacePipeline
from langchain.chains import RetrievalQA
from langchain.text_splitter import RecursiveCharacterTextSplitter

# Load documents and create vector store
text_splitter = RecursiveCharacterTextSplitter(chunk_size=1000, chunk_overlap=200)
documents = text_splitter.split_documents(your_documents)

embeddings = HuggingFaceEmbeddings(model_name="sentence-transformers/all-MiniLM-L6-v2")
vectorstore = Chroma.from_documents(documents, embeddings)

# Create QA chain
llm = HuggingFacePipeline.from_model_id(
    model_id="meta-llama/Llama-3-8B-Instruct",
    task="text-generation",
    model_kwargs={"device_map": "auto"}
)

qa_chain = RetrievalQA.from_chain_type(
    llm=llm,
    chain_type="stuff",
    retriever=vectorstore.as_retriever(search_kwargs={"k": 3}),
    return_source_documents=True
)

result = qa_chain.invoke({"query": "What is the capital of France?"})
print(result["result"])

Conversational QA

class ConversationalQA:
    def __init__(self, model, tokenizer):
        self.model = model
        self.tokenizer = tokenizer
        self.history = []
    
    def answer(self, question):
        context = "\n".join([f"Q: {q}\nA: {a}" for q, a in self.history])
        
        prompt = f"""Context:\n{context}

Q: {question}
A:"""
        
        inputs = self.tokenizer(prompt, return_tensors="pt").to(self.model.device)
        outputs = self.model.generate(**inputs, max_new_tokens=100)
        answer = self.tokenizer.decode(
            outputs[0][inputs.shape[-1]:], skip_special_tokens=True
        )
        
        self.history.append((question, answer.strip()))
        return answer.strip()

QA Challenges

Ambiguity

Questions can be ambiguous in multiple ways:

1. Lexical Ambiguity: "What is a bank?" (financial vs. river)
2. Structural Ambiguity: "I saw the man with the telescope" (who has the telescope?)
3. Referential Ambiguity: "Who is he?" (which person?)

Evidence Reasoning

Complex questions require reasoning over multiple pieces of evidence:

Calibration

Best Practices

Prompt Engineering

1. Clear instructions: "Answer based only on the provided context"
2. Format specification: "Provide a concise answer in one sentence"
3. Uncertainty handling: "If unsure, say 'I don't know'"
4. Citation requirements: "Cite the relevant passage"

Context Optimization

1. Relevant context: Retrieve the most relevant documents
2. Appropriate length: Balance context richness with model limits
3. Quality filtering: Remove noisy or irrelevant passages
4. Deduplication: Remove redundant information

Practice Exercises

1. Evaluation: Compare EM and F1 scores for a QA system on a dataset. When does EM give a misleading picture of performance?

2. Implementation: Build a simple RAG system using a vector store and LLM. Evaluate the impact of chunk size on answer quality.

3. Analysis: Analyze failure modes of a QA system on a benchmark dataset. What types of questions does it struggle with?

4. Research: Investigate the impact of retrieval quality on QA performance. How does the number of retrieved documents affect answer accuracy?

What to Learn Next