RandOpt

Neural Thickets: Diverse Task Experts Are Dense Around Pretrained Weights

Yulu Gan, Phillip Isola

Paper | Project Page | Openreview | Starting with a 1D Experiment:

News

• [2026-07] Iterative RandOpt is now on the iterative-randopt branch! Also support the pip install-able implementation that drops into existing verl or huggingface/trl setups.

Requirements

Option1: Python / Conda

(optional) conda activate your_env
pip install -r requirements.txt

Option2: Docker

From the directory containing RandOpt/:

Step	Command
Build	docker build -f RandOpt/docker/Dockerfile_vllm -t randopt-vllm:latest .
Run	docker run -it --gpus all randopt-vllm:latest bash
Run (with data)	docker run -it --gpus all -v /path/to/RandOpt/data:/workspace/data randopt-vllm:latest bash

Run RandOpt

Post-train on your own dataset

Please follow the instructions in CUSTOM_DATASET_GUIDE.md

Post-train on a standard dataset

First download the data here: data/README.md

Then, from the RandOpt directory:

Mode	Command
Single node	sbatch scripts/single_node.sh
Multiple nodes	sbatch scripts/multiple_nodes.sh
Local (no Slurm)	bash scripts/local_run.sh

Distill top-k models into a single model

Please follow the instructions in distillation/README.md.

Run Baselines

Please follow the instructions in baselines/README.md

Having questions?

Open an issue @ github.com/sunrainyg/RandOpt/issues.

Citation

@misc{gan2026neuralthickets,
      title={Neural Thickets: Diverse Task Experts Are Dense Around Pretrained Weights}, 
      author={Yulu Gan and Phillip Isola},
      year={2026},
      eprint={2603.12228},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/2603.12228}, 
}