Whiax/BERT-Transformer-Pytorch
Whiax/BERT-Transformer-Pytorch has 46 GitHub stars and 7 forks. GitHot measured 0 stars in 24 hours. Last checked 2026-11-15 05:22 UTC. Last push 2026-06-27.
Measured star growth
- in 24h
- 0
- in 7 days
- Collecting history
- in 15 days
- Collecting history
- in 1 month
- Collecting history
Net star changes over overlapping periods, not daily totals. Missing measurements are left blank. Stars measure attention, not software quality.
About
Basic implementation of BERT and Transformer in Pytorch in one short python file (also includes "predict next word" GPT task)
Machine LearningREADME.md
OriginalBERT-Transformer-Pytorch
Basic implementation of BERT and Transformer in Pytorch in one python file of ~300 lines of code (train.py).
This project aims to provide an easy-to-run easy-to-understand code for NLP beginners and people who want to know how Transformers work.
The project uses a simplified implementation of BERT (no labels are required for training).
The original implementation of Transformer uses an encoder and a decoder, here we only need the encoder.
The model can train in 30 minutes on 1 x RTX2070Super GPU.
Visualization of word embeddings:
Implementation details: https://hyugen-ai.medium.com/transformers-in-pytorch-from-scratch-for-nlp-beginners-ff3b3d922ef7
"Predict next word" task
August 2023 update:
- For experiment purposes, I also implemented the "predict next word" task which is used to train GPT.
- The code can be found in "main_predictnextword.py"
- This code is a slight modification of train.py