# Knowledge Distillation - Teacher model to help train the student model - Teacher is often pre trained - Student tries to imitate teacher - [Distillation Loss](Distillation%20Loss.md) - [Knowledge Distillation Survey 2021](Knowledge%20Distillation%20Survey%202021.md) - [Distilling the Knowledge in a Neural Network](Distilling%20the%20Knowledge%20in%20a%20Neural%20Network.md)