A assemble set of attention based toy model built from scratch
This repo may grow gradually and now it includes a tiny-transformer model, an attention-based LSTM model and some models reproduced such as Vison-Transformer and Swin-Transformer. Also, it includes an attention-based real-time segmentation model called Bisenet from CVPR2019 and another attention based model called VAN from arxiv 2022.
- 2022/10/17 Add MobileViT backbone, of which the trick is described at the head annotation.
- 2023/02/10 Add nanoGPT demo, a fantastic model from which to learn the popular chatgpt.
It contains a fundamental, essential and fancy autograd framework.