Reproduction of Vision Transformer in Tensorflow2. Train from scratch and Finetune

Tensorflow2 implementation of the Vision Transformer(ViT).

This repository is for An image is worth 16x16 words: Transformers for image recognition at scale and How to train your ViT? Data, Augmentation, and Regularization in Vision Transformers.

Limitations.

• Due to memory limitations, only the ti/16, s/16, and b/16 models were tested.
• Due to memory limitations, batch_size 2048 in s16 and 1024 in b/16 (in paper, 4096).
• Due to computational resource limitations, only reproduce using imagenet1k.

All experimental results and graphs are opend in Wandb.

Since this is personal project, it is hard to train with large datasets like imagenet21k. For a pretrain model with good performance, see the official repo. But if you really need it, contact me.