Projects
View on GitHub
sgugger/Adam-experiments
sgugger/Adam-experiments has 201 GitHub stars and 36 forks. GitHot measured 0 stars in 24 hours. Last checked 2026-11-16 20:16 UTC. Last push 2018-09-05.
Measured star growth
- in 24h
- 0
- in 7 days
- Update delayed
- in 15 days
- Update delayed
- in 1 month
- Update delayed
Net star changes over overlapping periods, not daily totals. Missing measurements are left blank. Stars measure attention, not software quality.
About
Experiments with Adam/AdamW/amsgrad
Developer ToolsREADME.md
OriginalExperiments with Adam
This repo contains the scripts used to perform the experiments in this blog post. If you're using this code or our results, please cite it appropriately.
You will find
- a script to train cifar10 to >94% accuracy in 30 epochs without Test Time Augmentation or 18 with.
- a script to finetune a pretrained resnet50 on the Standford cars dataset to 90% accuracy in 60 epochs.
- a script to train an AWD LSTM (or QRNN) to the same perplexity as the Saleforce team that created them but in just 90 epochs.
Requirements
- the fastai library is necessary to run all the models. If you didn't pip-install it, don't forget to have a simlink pointing to it in the directory where you clone this repo.
- pytorch 0.4.0 is necessary to have the implementation of amsgrad inside Adam.
- additonaly, the QRNNs model requires the cupy library.
To install
Run the script get_data.sh that will download and organize the data needed for the models
Experiments
Cifar10 dataset
- this should train cifar10 to 94.25% accuracy (on average):
python train_cifar10.py 3e-3 --wd=0.1 --wd_loss=False
- this should train cifar10 to 94% accuracy (on average) but faster:
python train_cifar10.py 3e-3 --wd=0.1 --wd_loss=False --cyc_len=18 --tta=True
Stanford cars dataset
- this should get 90% accuracy (on average) without TTA, 91% with:
python fit_stanford_cars.py '(1e-2,3e-3)' --wd=1e-3 --tta=True
Language models
- this should train an AWD LSTM to 68.7/65.5 perplexity without cache pointer, 52.9/50.9 with
python train_rnn.py 5e-3 --wd=1.2e-6 --alpha=3 --beta=1.5
- this should train an AWD QRNN to 69.6/66.7 perplexity without cache pointer, 53.6/51.7 with
python train_rnn.py 5e-3 --wd=1e-6 --qrnn=True