Whiax/BERT-Transformer-Pytorch
Whiax/BERT-Transformer-Pytorch has 46 GitHub stars and 7 forks. GitHot measured 0 stars in 24 hours. Last checked 2026-11-19 19:32 UTC. Last push 2026-06-27.
측정된 스타 증가
- in 24h
- 0
- in 7 days
- 기록 수집 중
- in 15 days
- 기록 수집 중
- in 1 month
- 기록 수집 중
서로 겹치는 기간의 순증가량이며 일별 합계가 아닙니다. 측정값이 없으면 비워 둡니다. 스타는 관심도를 나타내며 품질을 보장하지 않습니다.
소개
Basic implementation of BERT and Transformer in Pytorch in one short python file (also includes "predict next word" GPT task)
머신러닝README.md
OriginalBERT-Transformer-Pytorch
Basic implementation of BERT and Transformer in Pytorch in one python file of ~300 lines of code (train.py).
This project aims to provide an easy-to-run easy-to-understand code for NLP beginners and people who want to know how Transformers work.
The project uses a simplified implementation of BERT (no labels are required for training).
The original implementation of Transformer uses an encoder and a decoder, here we only need the encoder.
The model can train in 30 minutes on 1 x RTX2070Super GPU.
Visualization of word embeddings:

Implementation details: https://hyugen-ai.medium.com/transformers-in-pytorch-from-scratch-for-nlp-beginners-ff3b3d922ef7
"Predict next word" task
August 2023 update:
- For experiment purposes, I also implemented the "predict next word" task which is used to train GPT.
- The code can be found in "main_predictnextword.py"
- This code is a slight modification of train.py