I build models from the papers, then train them. Every folder here is code I wrote and ran rather than a fork of someone else's, so the notes next to it record what actually happened: hyperparameters, datasets, final losses, and the parts where my numbers disagree with the paper's.

What's in here

Folder	Contents
Replications/	37 papers reimplemented from scratch, each self-contained with a README of hyperparameters and results
SmolHub/	6 original small language models, 88M–200M parameters, with full training pipelines and HF Space deployments
Post-Training/	SFT and GRPO recipes, preference optimisation, plus my own runs on my own data

Replications

One folder per paper, from Bahdanau attention and the original Transformer through BERT, GPT, Llama, Mixtral and DeepSeek V3, plus GANs, VAEs, CLIP-style vision-language models and Whisper.

Browse the replications →

SmolHub

Small models built and trained end to end on hardware you can rent by the hour:

Model	Params	Architecture	Dataset
SmolLlama	~88M	Llama-style, RoPE + GQA	TinyStories
SmolMixtral	~124M	Mixture of Experts, 8 experts	TinyStories
SmolTransformer	~150M	Classic transformer + RMSNorm	FineWeb
StoryLlama	~88M	Llama variant, story-tuned	TinyStories
StoryMixtral	~200M	Mixtral variant	Story corpus
StoryKimi	~180M	DeepSeek V3-inspired MLA + MoE	Story corpus

These are original architectures rather than reproductions. Learned weights live on the Hub — for example StoryKimi and SmolMixtral — and the deployable Spaces are under Spaces.

Browse the small models →

Post-Training

Fine-tuning and post-training work, kept separate from the paper reproductions so the two don't get confused:

• sft/ — supervised fine-tuning recipes
• grpo/ — GRPO and RL-style post-training recipes
• runs/ — my own experiments on my own data, not reproductions of anyone's results

Browse the post-training experiments →

Deployed models

smolhub.com hosts the deployed models, and the Spaces themselves live on the HuggingFace account.

Usage

Every folder stands alone. Open its README for the hyperparameters and results, then run the notebook or script inside. Datasets marked as local are not committed; the folder README names the source. Weights and checkpoints are never committed — see .gitignore.

Contributing

Feel free to open issues or pull requests if you have suggestions, improvements, or additional replications to add!

Sponsors

Thank you everyone for their support and love for this project!

• abstrait
• Itz-Antaripa

🔗 Links

HuggingFace · Website · Deployed models

Authors

• @YuvrajSingh

Citation

If you find this repository useful in your research, please cite it:

@software{singh_paper_replications_2026,
  title     = {Paper Replications and Experiments},
  author    = {Singh, Yuvraj},
  year      = {2026},
  publisher = {Zenodo},
  doi       = {10.5281/zenodo.23198171},
  url       = {https://github.com/YuvrajSingh-mist/Paper-Replications-and-Experiments}
}

10.5281/zenodo.23198171 is the concept DOI: it always resolves to the newest archived version. This release, v1.1.0 (SmolHub merged in), is pinned at 10.5281/zenodo.23929573; the first release, v1.0.0, stays at 10.5281/zenodo.23198172.

License

• Code and notebooks: Apache License 2.0.
• Documentation, diagrams, generated results and published artifacts: CC BY 4.0.

Attribution: Yuvraj Singh. See NOTICE and CITATION.cff. SmolHub/ was merged in from a separate repository that declared MIT but shipped no licence file; it is distributed here under Apache-2.0.