--- toc: true title: DeiT tags: ['temp'] --- # DeiT - [paper](https://arxiv.org/abs/2012.12877) - [blog](https://ai.facebook.com/blog/data-efficient-image-transformers-a-promising-new-technique-for-image-classification/) - [Conv](Conv.md) free [Transformer](Transformer.md), [Vision Transformer](Vision%20Transformer.md) - does not require very large amount of data id:: 62a8a66a-941e-4a6d-918a-bb49cd496b15 - [Knowledge Distillation](Knowledge%20Distillation.md) - teacher-student strategy specific to transformers - [Distillation Token](Distillation%20Token.md) - ConvNet as teacher through [Attention](Attention.md) id:: 62a8a6b2-abf4-4869-934e-c75d05884304 - [ImageNet](ImageNet.md) - #+BEGIN_CAUTION Heh. Didnt they say no convs? #+END_CAUTION