Deep Learning Interview Questions for 5 Years Experience

CNN: applies learned filters over local receptive fields, translation-invariant, excels at spatial data (images). O(n) operations, fixed context size. RNN/LSTM: processes sequences step-by-step, maintains hidden state, good for sequential data but slow (sequential dependency) and struggles with long-range. O(n) sequential operations. Transformer: self-attention connects all positions directly, O(n²) memory but O(1) sequential depth, excellent long-range dependencies, fully parallelizable. Trend: transformers replaced RNNs for NLP entirely; Vision Transformers (ViT) are replacing CNNs for many vision tasks at scale.

Batch normalization normalizes layer activations to zero mean, unit variance across the batch, then scales/shifts with learned parameters γ and β. Benefits: (1) Reduces internal covariate shift — stable input distributions to each layer, (2) Allows higher learning rates — gradients don't explode, (3) Acts as mild regularizer — adds noise via batch statistics, (4) Reduces sensitivity to weight initialization. Placement: before or after activation (before is more common). Variants: LayerNorm (normalizes across features, not batch — used in transformers), GroupNorm (for small batches), InstanceNorm (style transfer).

Transformer components: (1) Input embeddings + positional encodings (sine/cosine or learned), (2) Multi-head self-attention: Q=XW_Q, K=XW_K, V=XW_V; attention = softmax(QK^T/√d_k)V; multiple heads capture different relationships, (3) Add & Norm: residual connection + LayerNorm after each sublayer — critical for training stability, (4) Feed-forward network: two linear layers with GELU activation — processes each position independently, (5) Encoder stacks these; decoder adds cross-attention to encoder output. Key design choices: residual connections prevent vanishing gradients; LayerNorm stabilizes training; positional encoding injects sequence order.

Classification: CrossEntropyLoss (multi-class), BCEWithLogitsLoss (binary/multi-label — numerically stable). Regression: MSELoss (when outliers are few), L1Loss/MAELoss (robust to outliers), HuberLoss (best of both). Object detection: Focal Loss for class imbalance, IoU loss for bounding boxes. Generative: adversarial loss (GANs), ELBO (VAEs), reconstruction + KL divergence. LLMs: CrossEntropy on next-token prediction. Tips: always use logit-space losses (CrossEntropyLoss not NLLLoss+softmax) for numerical stability; class weights for imbalanced data; label smoothing (0.1) to prevent overconfident predictions.

Transfer learning reuses a pretrained model's weights as initialization for a new task. Strategies by how different the target domain is: (1) Feature extraction — freeze all layers, only train new head (target data very small or similar domain), (2) Fine-tuning last few layers — unfreeze top layers, small LR, (3) Full fine-tuning — unfreeze all, very small LR (2e-5), (4) Progressive unfreezing — start by training only head, gradually unfreeze layers from top to bottom (ULMFiT approach). Key: use discriminative learning rates — lower LR for early layers (generic features), higher for later layers (task-specific features).

SGD with momentum: accumulates velocity in gradient direction, dampens oscillations, good for convex problems. Adam (Adaptive Moment Estimation): maintains per-parameter adaptive learning rates using first moment (mean) and second moment (variance) of gradients; default for deep learning. AdamW: Adam + decoupled weight decay (L2 regularization applied to weights, not adaptive gradients) — better generalization, preferred for transformers. Lion: sign-based optimizer, memory-efficient, recently popular for LLM pretraining. Learning rate schedule: warmup (0 → peak over first 10% steps) + cosine decay or linear decay.

A residual connection adds the input of a block directly to its output: output = F(x) + x, where F(x) is the learned transformation. Why critical: (1) Solves vanishing gradients in very deep networks — gradients can flow directly through skip connections without passing through many layers, (2) Makes optimization easier — the network only needs to learn the residual (difference) rather than a full transformation from scratch, (3) Allows training of 100+ layer networks (ResNet-152). Without residual connections, deep networks performed worse than shallow ones due to optimization difficulty — not overfitting.

Knowledge distillation trains a small "student" model to mimic a large "teacher" model's output probabilities (soft labels), not just the hard labels. Soft labels encode richer information — e.g., 0.85 dog, 0.10 wolf, 0.05 cat tells the student about feature relationships. Loss = α × CrossEntropy(student_hard, labels) + (1-α) × KLDiv(student_soft, teacher_soft) with temperature T to soften distributions. Use cases: (1) Deploy smaller model on mobile/edge, (2) Compress BERT → DistilBERT (40% smaller, 97% performance), (3) Ensemble distillation. For LLMs: distill GPT-4 outputs into a smaller model for a specific task.

Image augmentation: random horizontal/vertical flip, rotation (±15°), crop + resize, color jitter (brightness/contrast/saturation), cutout/cutmix/mixup, perspective transforms, elastic deformations. For medical imaging: conservative augmentation (no unnatural colors). Text augmentation: synonym replacement, random insertion/deletion, back-translation, EDA. Audio: time stretching, pitch shifting, noise addition, SpecAugment (mask frequency bands). Key principle: augmentations must preserve the label — flipping "6" to "9" in digit recognition changes the label! Albumentations library for image, torchvision.transforms for standard pipelines.

Bottleneck identification: torch.profiler shows CPU/GPU time per operation. Common bottlenecks: (1) Data loading — use DataLoader(num_workers=4, pin_memory=True, prefetch_factor=2), (2) GPU underutilization — increase batch size until GPU memory is ~80% full, (3) Float32 → use Automatic Mixed Precision (torch.cuda.amp.autocast()) for 2–3x speedup on modern GPUs, (4) Gradient accumulation — simulate larger batches without OOM, (5) torch.compile() (PyTorch 2.0) fuses operations for ~20% speedup, (6) Gradient checkpointing — trades compute for memory in very deep models. Always profile before optimizing.