Knowledge distillation creates small, deployable models that match large model accuracy. Core concept: train small student model to mimic soft outputs (probability distributions) of large teacher model, not just hard labels. Soft targets contain more information - teacher assigns 0.7 probability to "cat", 0.2 to "leopard", encoding similarity structure. Training: loss = alpha * cross_entropy(student, hard_labels) + (1-alpha) * KL_divergence(student, teacher_soft_targets) where teacher is run at higher temperature T to soften distributions. Types: 1) Task-specific distillation: DistilBERT distills BERT to 40% size while retaining 97% of GLUE performance. TinyBERT uses additional intermediate layer matching for better knowledge transfer. 2) Data-free distillation: generate synthetic data by inverting the teacher model - useful when training data is proprietary. 3) LLM distillation: fine-tune small open source model (3B) on GPT-4o generated examples for specific tasks - creates task-specific specialist that outperforms the small base model significantly. Implementation: use Hugging Face Trainer with custom distillation loss. Teacher and student share tokenizer. Practical results: 12-layer BERT distilled to 6-layer achieves 97% performance at 50% speed. GPT-4o distilled task specialist on specific domain can match 90%+ of GPT-4o performance at 100x lower cost.