Adding Vision Transformer to torchvision/models

[yiwen-song]

🚀 The feature

1. Adding ViT architecture from this paper: "An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale"
2. Adding DeiT architecture from this paper: "Training data-efficient image transformers & distillation through attention"

@fmassa @datumbox @mannatsingh @kazhang

Motivation, pitch

Vision Transformer models should exist in torchvision repo because they are good models :)

I'm currently working on this project.

Additional context

We can also consider adding some techniques from the following papers ^^
For example, adding Conv stem for ViT, see details in "Early Convolutions Help Transformers See Better"

References:
https://github.com/google-research/vision_transformer
https://github.com/facebookresearch/deit
https://github.com/facebookresearch/ClassyVision/blob/main/classy_vision/models/vision_transformer.py

cc @datumbox

[mannatsingh]

Note that DeiT has the same architecture as ViT - it's the same model, only the training setup is different!

[yiwen-song]

Note that DeiT has the same architecture as ViT - it's the same model, only the training setup is different!

Interesting! I was looking at the implementation here
https://github.com/facebookresearch/deit/blob/main/models.py#L20
and found that DeiT model actually inherits the ViT model class.

Does it make sense if I also do this in torchvision?

[datumbox]

@sallysyw First of all, thanks for adding this. This is awesome.

Concerning DeIT, we have a proposal to support distillation tasks on the future and there is an on going RFC that will allow you to have different pre-trained weights for the same architecture. So if you want to pursue that route, we should be able to accommodate.

[mannatsingh]

Oh yeah, I meant the baseline (no-distillation) DeIT is the same as ViT. Supporting the distillation token + workflow is your call :)

[take2rohit]

I believe that having ViT/DeiT in torchvision library would be really useful!
So is anyone else working with the implementation or should I go ahead and create a PR?

[datumbox]

@take2rohit Thanks. @sallysyw is working on it at PR #4594.