Projects
View on GitHub
kingoflolz/swarm-jax
kingoflolz/swarm-jax has 241 GitHub stars and 22 forks. GitHot measured 0 stars in 24 hours. Last checked 2026-11-19 12:33 UTC. Last push 2023-05-12.
Measured star growth
- in 24h
- 0
- in 7 days
- Collecting history
- in 15 days
- Collecting history
- in 1 month
- Collecting history
Net star changes over overlapping periods, not daily totals. Missing measurements are left blank. Stars measure attention, not software quality.
About
Swarm training framework using Haiku + JAX + Ray for layer parallel transformer language models on unreliable, heterogeneous nodes
Developer ToolsREADME.md
OriginalPipelined Swarm Training
Swarm training "framework" using Haiku + Jax + Ray.
Designed for training large language models in a model parallel fashion with unreliable, heterogeneous nodes. (eventually)
Look in swarm_run.py for an example of running a character transformer on enwik8.
TODOs
- Forward passes
- Backward passes with activation reconstruction
- Run optimizer
- Logging
- Checkpointing
- Actually do pipelining
- fp16 with static loss scaling
- Integer quantization for activations and gradients between layers
- Get rid of pipeline stalls from running optimizer
- Data parallelism with multiple nodes per layer and gradient/weight aggregation
- Heterogeneous nodes with potentially multiple layers per node
- Handle unbalanced and unreliable nodes (layerdrop)
- Dynamic node addition
- 1T or bust?