Knowledge distillation lets you train efficient small LLMs by transferring expertise from large models.

Knowledge distillation transforms large language models into smaller, efficient versions by guiding them to mirror a teacher’s refined output patterns.

Rather than starting from random weights, a student model learns directly from the teacher’s nuanced responses. This avoids the cost of training on raw data alone, instead leveraging the teacher’s probability distributions—such as a 90% "cat," 8% "dog," and 2% "car" classification—where each percentage reflects subtle feature relationships. The student absorbs these finer distinctions through structured guidance.

The core adjustment lies in the loss function, which aligns the student’s logits with the teacher’s using a temperature-scaled softmax. By dividing logits by a temperature value before applying softmax, the teacher’s uncertainty becomes explicit, shaping the student’s learning. The divergence between student and teacher is then quantified with Kullback-Leibler divergence, adjusted by temperature² to balance precision:

def distillation_loss(student_logits, teacher_logits, temperature):
    soft_targets = F.softmax(teacher_logits / temperature, dim=1)
    soft_predictions = F.log_softmax(student_logits / temperature, dim=1)
    loss = F.kl_div(soft_predictions, soft_targets, reduction='batchmean') * (temperature ** 2)
    return loss

This method enables production-ready models with critical advantages: faster inference, lower GPU costs (a 7B distilled model runs on a single T4 GPU without API fees), and on-premise deployment for privacy-sensitive workloads. While the student model won’t surpass the teacher’s accuracy, it retains 90% of its performance for specialized tasks—like code generation or sentiment analysis—while occupying just 10% of the original size. For targeted applications, distillation delivers a lightweight alternative to relying on oversized models.