DiffUTE

This repository is the code of our NeurIPS'23 paper "DiffUTE: Universal Text Editing Diffusion Model".

:tada::tada::tada: We have reproduced our method using the AnyText dataset, and the weights can be downloaded at the URL. This checkpoint can be load for a demo.

Getting Started with DiffUTE

Installation

The codebases are built on top of diffusers. Thanks very much.

Requirements

• Linux or macOS with Python ≥ 3.8
• PyTorch ≥ 1.10.0 and torchvision that matches the PyTorch installation. You can install them together at pytorch.org to make sure of this
• OpenCV
• transformers

Steps

1. Install diffusers following https://github.com/huggingface/diffusers.

2. Prepare datasets. Due to data sensitivity issues, our data will not be publicly available now, you can reproduce it on your own data, and all images with text are available for model training. Because our data is present on Ali-Yun oss, we have chosen pcache to read the data we have stored. You can change the data reading method according to the way you store the data.

3. Train VAE

4. Train DiffUTE

Experimental results

Citing DiffUTE

If you use DiffUTE in your research or wish to refer to the baseline results published here, please use the following BibTeX entry.

@inproceedings{DiffUTE,
      title={DiffUTE: Universal Text Editing Diffusion Model},
      author={Chen, Haoxing and Xu, Zhuoer and Gu, Zhangxuan and Lan, Jun and Zheng, Xing and Li, Yaohui and Meng, Changhua and Zhu, Huijia and Wang, Weiqiang},
      booktitle={Thirty-seventh Conference on Neural Information Processing Systems (NeurIPS)},
      year={2023}
}

Contacts

Please feel free to contact us if you have any problems.

Email: [email protected] or [email protected]