Keyphrase Extraction: Identifying Important Concepts

Module: Natural Language Processing | Difficulty: Advanced

TF-IDF

TextRank

Supervised

Evaluation

| Model | F1@5 | F1@10 | |-------|------|-------| | TF-IDF | 25.3 | 31.2 | | TextRank | 28.1 | 35.6 | | BERT-based | 35.2 | 42.1 |

import numpy as np
from sklearn.feature_extraction.text import TfidfVectorizer

def extract_keyphrases(text, top_k=10):
    vectorizer = TfidfVectorizer(ngram_range=(1, 3), stop_words='english')
    tfidf_matrix = vectorizer.fit_transform([text])
    feature_names = vectorizer.get_feature_names_out()
    scores = tfidf_matrix.toarray()[0]
    top_indices = np.argsort(scores)[::-1][:top_k]
    return [(feature_names[i], scores[i]) for i in top_indices]

Research Insight: Neural keyphrase extraction models outperform traditional methods by 10-15% F1 because they capture context better. The key challenge is handling keyphrases of varying lengths and specificity levels.