Weistrass/DyRef
Weistrass/DyRef has 370 GitHub stars and 9 forks. GitHot measured 0 stars in 24 hours. Last checked 2026-11-17 02:15 UTC. Last push 2026-08-05.
What you can do
- We introduce OmniRef-Bench to evaluate complex multi-reference image generation (MRIG) across diverse combinations of reference-image types and quantities. Our evaluation reveals that current open-source models still struggle substantially on this challenging benchmark.
- We develop an automated data synthesis pipeline for complex MRIG and propose DyRef to address the sharp performance degradation of existing open-source models as reference complexity increases. Trained with DyRef, Qwen-Image-Edit-2511 achieves performance comparable to the closed-source model Nano Banana Pro and surpasses Seedream 4.5.
- DyRef consistently enhances open-source models, including Qwen-Image-Edit-2511 and FLUX.2-klein-base, on MRIG benchmarks (OmniRef-Bench, MultiBanana, and OmniContext) while also improving single-image editing performance on ImgEdit and DreamBench++, without compromising the foundation models' other capabilities.
Measured star growth
- in 24h
- 0
- in 7 days
- Update delayed
- in 15 days
- Update delayed
- in 1 month
- Update delayed
Net star changes over overlapping periods, not daily totals. Missing measurements are left blank. Stars measure attention, not software quality.
About
[ECCV 2026 Oral] Official repository for Scaling Multi-Reference Image Generation with Dynamic Reward Optimization
Creative & MediaREADME.md
OriginalScaling Multi-Reference Image Generation with Dynamic Reward Optimization
Wenwang Huang1,*, Yusen Fu1,*, Junjie Wang1, Mengfei Huang1, Yulin Li1, Gan Liu2, Jing Cai2, Yancheng He2, Zhuotao Tian1,3,†
1 Harbin Institute of Technology, Shenzhen,
2 Independent Researcher,
3 Shenzhen Loop Area Institute,
* Equal contribution · † Corresponding author
🎬 Demo Video
https://github.com/user-attachments/assets/cffc168a-a785-47be-baf2-4ef504ae9f22
🔖 Table of Contents
- News
- Todo List
- Highlights
- Results Gallery
- Motivation
- Method
- Installation
- Quickstart
- Evaluation
- Acknowledgement
- Contact
- Citation
🔥News
- [2026.06.18] Our paper has been accepted by ECCV 2026 🎉🎉🎉. Congratulations to all collaborators! 🎊🎊🎊
- [2026.07.02] We have released the training and inference code of DyRef, along with the corresponding training data, benchmark, and model weights. Feel free to use it! ⭐⭐⭐
- [2026.07.24] 🍾Our paper has been selected as an Oral Presentation at ECCV 2026. We are grateful to the program committee and reviewers, and excited to present DyRef to the community!
📋Todo List
- Release our paper on arXiv.
- Open-source the training code of DyRef for Qwen-Image-Edit-2511, and Flux2-klein-base.
- Release the training data used in our work.
- Open-source OmniRef-Bench, our proposed benchmark.
- Release the model weights trained by our method.
- Improve code efficiency and structure.
- Release the project page
✨Highlights

We introduce OmniRef-Bench to evaluate complex multi-reference image generation (MRIG) across diverse combinations of reference-image types and quantities. Our evaluation reveals that current open-source models still struggle substantially on this challenging benchmark.
We develop an automated data synthesis pipeline for complex MRIG and propose DyRef to address the sharp performance degradation of existing open-source models as reference complexity increases. Trained with DyRef, Qwen-Image-Edit-2511 achieves performance comparable to the closed-source model Nano Banana Pro and surpasses Seedream 4.5.
DyRef consistently enhances open-source models, including Qwen-Image-Edit-2511 and FLUX.2-klein-base, on MRIG benchmarks (OmniRef-Bench, MultiBanana, and OmniContext) while also improving single-image editing performance on ImgEdit and DreamBench++, without compromising the foundation models' other capabilities.
In addition, using our proposed automated data construction pipeline, we evaluated the model using training datasets containing approximately 5K, 7.5K, and 14K samples. We observed consistent performance gains as the training dataset size increased, with no clear evidence of performance saturation. However, due to limited computational resources, we did not evaluate larger training datasets.
🖼️Results Gallery
In this part, we present qualitative comparisons between our method and existing approaches across various combinations of reference types. The prompts shown in the figures are abbreviated versions for visualization purposes and are not the actual prompts used during inference. In addition, we use color to indicate the attributes specified in the user prompt and their corresponding reference images.


💡Motivation

In this work, we observe that the performance of current mainstream open-source image generation models degrades substantially as the number of reference images increases and the diversity of reference types expands. This limitation significantly restricts their potential for professional applications, such as creative image generation and video keyframe generation.
🧩Method

Illustration of DyRef. To address this issue, we propose DyRef, a two-stage training framework.
In Stage I, we employ SFT to equip the model with the basic capability to handle complex MRIG tasks. In Stage II, (a) DRS enlarges the reward differences across samples for better training, and (b) DAR enhances the model’s focus on samples with numerous mixed-type reference images.
⚙️Installation
DyRef uses separate Conda environments for SFT, RL, and benchmark evaluation because each component has its own dependency requirements.
SFT Environment
conda create -n dyref_sft python=3.11 -y
conda activate dyref_sft
cd DyRef/sft
pip install -e .
RL Environment
conda create -n dyref_rl python=3.11 -y
conda activate dyref_rl
cd DyRef/rl
pip install -e .[deepspeed]
Model Download
Please download the following models from Hugging Face:
- Qwen/Qwen-Image-Edit-2511
- google/siglip2-base-patch16-384
- facebook/dinov2-base
- openai/clip-vit-large-patch14
- black-forest-labs/FLUX.2-klein-base-9B (if you need)
For example:
hf download Qwen/Qwen-Image-Edit-2511 --local-dir "your path to save this model"
🚀Quickstart
Model Inference
Before running the following command, please download the Qwen-Image-Edit-2511 model and the pretrained LoRA weights we provide.
conda activate dyref_sft
python model_inference/Qwen-Image-Edit-2511.py
Train
If you would like to train a model from scratch using your own data or reproduce our training pipeline, please run the following command.
1. Stage 1: SFT Training
conda activate dyref_sft
cd DyRef/sft
bash all_scripts/Qwen-Image-Edit-2511_lora.sh
2. Convert SFT Weights for RL
Since the original repositories used for SFT and RL training adopt different formats for storing LoRA weights, the following conversion script is required to ensure that the LoRA weights generated during the SFT stage can be correctly loaded during RL training.
cd DyRef/sft
python all_scripts/diffusers_peft_transfer.py --mode d2p \
--input /path/to/sft_checkpoint.safetensors \
--output /path/to/output_peft_format_dir \
--prefix transformer \
--verify
3. Stage 2: RL Training
conda activate dyref_rl
cd DyRef/rl
bash scripts/qwen2511-gdpo-rank64-add2k5-csd-siglipv2_flat-sigmoid0.65-focal_loss.sh
📊Evaluation
OmniRef-Bench is designed to measure whether generated images preserve and combine multiple references in a balanced way.
The benchmark covers:
- Subject fidelity
- Style consistency
- Background consistency
- Lighting consistency
- Pose consistency
- Overall multi-reference alignment
The evaluation-related code is organized under:
benchmark/Grounded-SAM-2_patch/benchmark/CSD_patch/benchmark/AlphaPose_patch/benchmark/MLLM_eval/
1. Convert RL Weights for Evaluation
cd DyRef/sft
python all_scripts/diffusers_peft_transfer.py --mode p2d \
--input /path/to/rl_checkpoint_dir \
--output /path/to/output.safetensors \
--prefix '' \
--verify
2. Generate Images for Evaluation
conda activate dyref_sft
cd DyRef/sft
bash all_scripts/eval/eval_ourbench_lora_2511.sh
3. Run OmniRef-Bench
cd benchmark
bash eval_suite.sh \
/path/to/generated_images \
/path/to/output_dir \
/path/to/test_set \
/path/to/test_set.json
For more details, see benchmark/README.md.
🙏Acknowledgement
DyRef is built on several excellent open-source projects:
- DiffSynth-Studio for SFT training infrastructure
- Flow-Factory for RL training infrastructure
- Grounded-SAM-2 for subject and background evaluation
- CSD for style consistency evaluation
- AlphaPose for pose evaluation
📧Contact
If you have any questions, suggestions, or ideas, please feel free to contact us at [email protected].
📝Citation
If you find DyRef useful in your research, please consider citing it:
@article{huang2026scaling,
title={Scaling Multi-Reference Image Generation with Dynamic Reward Optimization},
author={Huang, Wenwang and Fu, Yusen and Wang, Junjie and Huang, Mengfei and Li, Yulin and Liu, Gan and Cai, Jing and He, Yancheng and Tian, Zhuotao},
journal={arXiv preprint arXiv:2606.26947},
year={2026}
}