NLP Interview Questions for 5 Years Experience

Word2Vec (2013): trains a shallow neural network to predict context words; produces context-independent static embeddings. Two variants: CBOW (predict word from context) and Skip-gram (predict context from word). GloVe: uses global word co-occurrence statistics instead of local context windows — better at capturing word analogies. FastText: extends Word2Vec with subword (character n-gram) information — handles morphologically rich languages and out-of-vocabulary words. All three produce static embeddings — BERT contextual embeddings supersede them for most tasks.

Attention allows a model to weigh how relevant each input token is when producing each output token — instead of compressing the entire input into one fixed vector (RNN bottleneck). Scaled dot-product attention: score = softmax(QK^T / sqrt(d_k))V. Self-attention lets each token attend to all other tokens in the same sequence. Multi-head attention runs attention in parallel with different learned projections. This is why transformers can capture long-range dependencies that RNNs struggled with, and why BERT understands "bank" differently in "river bank" vs "bank account".

BERT (Bidirectional Encoder Representations from Transformers): encoder-only transformer pretrained with Masked Language Modeling (MLM) — predicts masked tokens using both left and right context. Best for classification, NER, QA — understanding tasks. GPT: decoder-only transformer pretrained with Causal Language Modeling (CLM) — predicts next token using only left context. Best for text generation. BERT = read the full sentence; GPT = write the next word. Modern alternatives: RoBERTa (better BERT training), DeBERTa (adds disentangled attention), T5 (encoder-decoder, treats everything as text-to-text).

Steps: (1) Choose base model (bert-base-uncased for English classification), (2) Add classification head: linear layer on top of [CLS] token, (3) Tokenize dataset with the model's tokenizer (same vocab!), (4) Fine-tune with low learning rate (2e-5 to 5e-5) — too high destroys pretrained weights, (5) Train for 3–5 epochs with warmup schedule, (6) Evaluate with F1 (not accuracy for imbalanced classes). Hugging Face Trainer handles most boilerplate. Key pitfalls: label leakage in preprocessing, using wrong tokenizer, not setting model.train() mode.

NER identifies and classifies named entities (persons, organizations, locations, dates) in text. Implementation: (1) Token classification task — each token gets a label (B-PER, I-PER, O using BIO scheme), (2) Fine-tune BERT with a token classification head, (3) Post-process BIO tags to extract entities. Evaluation: use entity-level F1 (not token-level). Production: spaCy (fast, rule + ML), Flair (good accuracy), Hugging Face transformers. Common challenges: nested entities, domain-specific entities (medical, legal) needing custom training data.

Techniques: (1) Oversampling minority class — random duplication or EDA (Easy Data Augmentation: synonym replacement, random insertion/deletion), (2) Undersampling majority class, (3) Class weights in loss function: CrossEntropyLoss(weight=torch.tensor([1.0, 5.0])) for 5x minority, (4) Focal loss — reduces weight of easy examples, forces model to focus on hard minority cases, (5) Threshold tuning — optimize threshold on validation set using F1 not accuracy, (6) Data augmentation with back-translation (translate to French, back to English). Always stratify your train/val/test splits.

Semantic similarity measures how close two sentences are in meaning. Approaches: (1) Sentence-BERT (SBERT) — fine-tunes BERT with siamese network on NLI pairs, produces sentence embeddings; cosine similarity gives semantic score, (2) Cross-encoder — concatenates two sentences, outputs similarity score; more accurate but O(n²) for large datasets, (3) Bi-encoder (SBERT) for retrieval (fast), cross-encoder for reranking top-k results. Production pattern: FAISS index with SBERT embeddings for fast approximate search, then cross-encoder reranker for top-10. Used in: duplicate detection, semantic search, FAQ matching.

Extractive summarization selects the most important existing sentences from the document — no new text generated. Methods: TextRank (graph-based sentence ranking), BERTSum (BERT for extractive scoring). Fast, faithful, but stilted output. Abstractive summarization generates new text that captures the meaning — like a human summary. Methods: BART, T5, Pegasus (pretrained for summarization). Better quality but can hallucinate. Hybrid: extract key sentences, then abstractively compress them. Evaluation: ROUGE-1/2/L (n-gram overlap), BERTScore (semantic similarity). For legal/medical: prefer extractive (faithfulness critical).

Coreference resolution identifies which mentions in text refer to the same entity — "Elon Musk founded Tesla. He is also CEO of SpaceX." (He = Elon Musk). Hard because: (1) Pronouns are ambiguous — "The trophy wouldn't fit in the bag because it was too big" (it = trophy), (2) Long-distance dependencies, (3) World knowledge required. Modern approaches: SpanBERT (spans as entities), e2e-coref (end-to-end neural). Critical for: multi-document QA, dialogue systems, information extraction. Often an unsolved bottleneck in production NLP pipelines.

Task-specific metrics: Classification — F1 (macro for balanced, weighted for imbalanced), Matthews Correlation Coefficient for binary. NER — entity-level F1 (seqeval library). Translation/Summarization — BLEU (n-gram precision), ROUGE (recall-oriented). Generation — BERTScore (contextual F1), BLEURT (learned metric). QA — Exact Match, F1 overlap. Beyond automated metrics: (1) Human evaluation for quality, coherence, factuality, (2) Adversarial testing — check model on edge cases, negations, typos, (3) Behavioral testing (CheckList) — test linguistic capabilities systematically. Never deploy based on a single metric.

In RNNs, gradients are multiplied repeatedly through time during BPTT. If weights < 1, gradients vanish to zero — early timesteps get no learning signal (model forgets long-range dependencies). If weights > 1, gradients explode — unstable training. LSTM solution: add a cell state (memory highway) with forget, input, and output gates that control information flow using sigmoid activations (0–1 range). The additive cell state update (not multiplicative) allows gradients to flow without decay. Transformers bypassed this entirely with attention — no sequential processing, no gradient propagation through time.

Production pipeline: (1) Preprocessing — standardize encoding, handle HTML/emoji, language detection, (2) Model serving — ONNX-exported model on FastAPI, batching requests for throughput, (3) Monitoring — log predictions + confidence scores, (4) Data drift detection — track input text distributions (embedding centroid shift, vocabulary drift), (5) Label drift — monitor prediction distribution changes over time with Evidently or NannyML, (6) Feedback loop — collect corrections, add to training set, retrain monthly, (7) Canary deployment — test new model on 5% traffic before full rollout. Model performance degrades silently in production without monitoring.