Skip to content
프로젝트

hayeong0/Diff-HierVC has 237 GitHub stars and 20 forks. GitHot measured 0 stars in 24 hours. Last checked 2026-11-21 10:28 UTC. Last push 2024-07-03.

이 수치를 측정하는 방법

측정된 스타 증가

in 24h
0
in 7 days
기록 수집 중
in 15 days
기록 수집 중
in 1 month
기록 수집 중

서로 겹치는 기간의 순증가량이며 일별 합계가 아닙니다. 측정값이 없으면 비워 둡니다. 스타는 관심도를 나타내며 품질을 보장하지 않습니다.

소개

Official Pytorch Implementation of "Diff-HierVC: Diffusion-based Hierarchical Voice Conversion with Robust Pitch Generation and Masked Prior for Zero-shot Speaker Adaptation"

머신러닝

README.md

Original

Diff-HierVC: Diffusion-based Hierarchical Voice Conversion with Robust Pitch Generation and Masked Prior for Zero-shot Speaker Adaptation

The official Pytorch implementation of Diff-HierVC (Interspeeh 2023, Oral)

Ha-Yeong Choi, Sang-Hoon Lee, Seong-Whan Lee

image

Overall architecture

Although voice conversion (VC) systems have shown a remarkable ability to transfer voice style, existing methods still have an inaccurate pitch and low speaker adaptation quality. To address these challenges, we introduce Diff-HierVC, a hierarchical VC system based on two diffusion models. We first introduce DiffPitch, which can effectively generate $F_0$ with the target voice style. Subsequently, the generated $F_0$ is fed to DiffVoice to convert the speech with a target voice style. Furthermore, using the source-filter encoder, we disentangle the speech and use the converted Mel-spectrogram as a data-driven prior in DiffVoice to improve the voice style transfer capacity. Finally, by using the masked prior in diffusion models, our model can improve the speaker adaptation quality. Experimental results verify the superiority of our model in pitch generation and voice style transfer performance, and our model also achieves a CER of 0.83% and EER of 3.29% in zero-shot VC scenarios.

🎧 Audio Demo

https://diff-hiervc.github.io/audio_demo/

📑 Pre-trained Model

Our model checkpoints can be downloaded here.

  • model_diffhier.pth
  • voc_hifigan.pth
  • voc_bigvgan.pth

🔨 Usage

  1. Clone this rep && Install python requirement
git clone https://github.com/hayeong0/Diff-HierVC.git
pip install -r req*
  1. Download the pre-trained model checkpoint from drive and place it in the following path.
.
├── ckpt
│   ├── config.json
│   └── model_diffhier.pth ✅
├── inference.py
├── infer.sh
├── model
├── module
├── requirements.txt
├── utils
└── vocoder
    ├── hifigan.py
    ├── modules.py
    └── voc_hifigan.pth ✅
    └── voc_bigvgan.pth ✅
  1. Run infer.sh

diffpitch_ts refers to the time step of the pitch generator and diffvoice_ts refers to the time step of the Mel generator.

Empirically, it has been observed that if the time step of diffpitch is too small, noise remains, and if it is too large, excessive diversity occurs.

Please use it appropriately for your dataset!

bash infer.sh

python3 inference.py \
    --src_path './sample/src_p241_004.wav' \
    --trg_path './sample/tar_p239_022.wav' \
    --ckpt_model './ckpt/model_diffhier.pth' \
    --ckpt_voc './vocoder/voc_bigvgan.pth' \
    --output_dir './converted' \
    --diffpitch_ts 30 \
    --diffvoice_ts 6

🎧 Test it on your own dataset and share your interesting results! :)

🎓 Citation

@inproceedings{choi23d_interspeech,
  author={Ha-Yeong Choi and Sang-Hoon Lee and Seong-Whan Lee},
  title={{Diff-HierVC: Diffusion-based Hierarchical Voice Conversion with Robust Pitch Generation and Masked Prior for Zero-shot Speaker Adaptation}},
  year=2023,
  booktitle={Proc. INTERSPEECH 2023},
  pages={2283--2287},
  doi={10.21437/Interspeech.2023-817}
}

💎 Acknowledgements

License

This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.

CC BY-NC-SA 4.0