Koifish

Sparse & quantized LLM training/inference/CPT/SFT/DPO.

Koifish is a lightweight & powerfule framework in C++/cu, which needs much less resource than other. It needs only sveral hours to fintue 4~32B model on single 4090. It needs only one day to train ~2B model on single 4090 as shown in the following table.

Model	Parameter	Loss(Baseline)	GPU Memory	Total Time (Training+Evaluating)	Throughput	Log
GPT2-124M	124M	3.293(3.425)	~6.8G	~8 hours	~140k/s	log
GPT2-774M	774M	3.146(3.00)	~15G	~18 hours	~70k/s	log
GPT2-1558M	1558M	3.04(2.83)	~23G	~30 hours	~50k/s	log

Note

1. The baseline results are from GPT2. Our training only used 5B tokens of FineWeb.
2. The evaluating time depends on the frequency of testing and the sampling ratio(We use only ~10% randomly sampled tokens to reduce total time).

Features

• Support both Masked diffusion language models (MDLMs) and AR models
• Hybrid 16/8/4/2/1 bit training
• Training Qwen3-0.6B on single 4090
• Evolutionary optimization of experts
• Inference of QWen3-32B on single 4090
• Automatic detection of training instability
• Json config file
• Self-contained C++/cu project with minimal dependencies

Minimum dependencies

The inference module (cpu version) does not require any third-party libraries.

Training module needs more:

• cudnn(would removed in future version)
• GPUs that support FP8 computation (Compute Capability 8.9 or higher) with more than 16GB+ VRAM
• CUDA Toolkit (12.5+)

Download & Build

# sudo apt-get -y install libcudnn9-dev-cuda-12  # maybe need this to install CUDNN
# sudo apt-get install libicu-dev
# export CPATH=/usr/local/cuda/include:$CPATH        # maybe need this to export CPATH
git clone --recurse-submodules https://github.com/gruai/koifish
cd koifish
mkdir build && cd build && cmake ..
make clean && make -j$(nproc)

Tutorial

1. Training of QWen3_0.6B on single 4090
2. Training of GPT2_1558M_ on single 4090
3. Training of GPT2(774M/124M) on single 3090

Techniques/Tricks

Working plan

• Diffusion Language Model
• High accuracy 1-bit training/inference algorithm

Citation

Please use the following bibtex entry:

@article{chen2025eoe,
  title={EOE: Evolutionary Optimization of Experts for Training Language Models},
  author={Chen, Yingshi},
  journal={arXiv preprint arXiv:2509.24436},
  year={2025}
}

Contributing

• Any help with managing issues, PRs and projects is very appreciated!

Acknowledgements

• Thanks very much for the highly instructive work of qwen600, calm, llm.c & ggml.
• Thanks very much for the support of Professor TianLan.

More

History

QQ group: 167653113