Skip to content
Projects

AaronZ345/ISDrama has 239 GitHub stars and 0 forks. GitHot measured 0 stars in 24 hours. Last checked 2026-11-15 23:38 UTC. Last push 2026-11-09.

How these counts are measured

Measured star growth

in 24h
0
in 7 days
Collecting history
in 15 days
Collecting history
in 1 month
Collecting history

Net star changes over overlapping periods, not daily totals. Missing measurements are left blank. Stars measure attention, not software quality.

About

Dataset and evaluation code of ISDrama(ACM-MM 2025): Immersive Spatial Drama Generation through Multimodal Prompting

Creative & Media

README.md

Original

ISDrama: Immersive Spatial Drama Generation through Multimodal Prompting
ACM Multimedia 2025

Yu Zhang*, Wenxiang Guo*, Changhao Pan*, Zhiyuan Zhu*, Tao Jin, Zhou Zhao Zhejiang University

Paper Demo Page Dataset 语音之家 PaperWeekly 知乎

Overview

ISDrama introduces multimodal spatial drama generation and the MRSDrama recorded dataset. This repository provides dataset access and spatial evaluation tools. MRSDrama and MRSAudio are publicly released training resources for spatial audio generation and understanding.

For training, see the datasets and data format. For paired audio metrics, see Quick start and the evaluation guide. To cite this work, use the BibTeX entry.

News

  • 2025.07 · Released the evaluation code.
  • 2025.07 · Released the full MRSDrama dataset.
  • 2025.07 · ISDrama was accepted to ACM Multimedia 2025.

Quick start

Resource Available here
Training data MRSDrama and MRSAudio download links
Evaluation Paired spatial audio feature extraction and metrics
Encoder Published BAT checkpoint, verified before loading
Runtime Python 3.8 to 3.10; PyTorch 2.0.1; CPU or an existing CUDA device

1. Install the environment

git clone https://github.com/AaronZ345/ISDrama.git
cd ISDrama
conda env create -f environment.yml
conda activate isdrama_eva
bash timm_patch/patch.sh

2. Prepare the encoder and paired audio

Download the published BAT checkpoint to ckpt/finetuned.pth. Its SHA-256 is checked before a private copy is loaded; custom or modified checkpoints are rejected.

Place ground-truth WAV files in data/gt and generated WAV files in data/infer with identical filenames. Each file must be stereo. Choose an output directory that does not exist.

3. Run spatial evaluation

bash evaluate/eval.sh --gt-dir data/gt --pred-dir data/infer \
  --checkpoint ckpt/finetuned.pth --output-dir outputs/run-1 --device cpu

The runner extracts BAT embeddings and IPD/ILD features, then reports mean absolute errors and distance/azimuth cosine similarities. It reserves the output directory and removes partial output if a stage fails. For CUDA, use --device cuda and select an existing GPU with CUDA_VISIBLE_DEVICES.

To evaluate IPD and ILD without the encoder checkpoint:

bash evaluate/eval.sh --gt-dir data/gt --pred-dir data/infer \
  --output-dir outputs/ipd-ild-run-1 --skip-embeddings

See the evaluation guide for metric definitions, numeric feature files, historical normalization, and evaluation of existing features.

Training datasets

Dataset Training resources Access
MRSDrama Binaural drama, scripts, video, geometric poses, and text prompts Hugging Face · Google Drive
MRSAudio MRSLife, MRSSpeech, MRSSing, and MRSMusic with spatial recordings and multimodal annotations Hugging Face

Prepare either dataset for your model's training pipeline and follow its own data card and license. The paired ground-truth/generated directories above are evaluation inputs. The encoder's BAT preprocessing profile does not select a training dataset.

Using MRSDrama means accepting its dataset license. Its data-format guide describes recordings, annotations, and listener-centric geometric sequences.

MRSDrama data collection

MRSDrama data collection from scripts and binaural recording through alignment, pose annotation, quality checks, and segmentation.
Figure 2. How MRSDrama was recorded, annotated, and prepared.

Evaluation outputs

Score Definition
IPD MAE Error between aligned interaural phase-difference features
ILD MAE Error between aligned interaural level-difference features
Distance cosine similarity Similarity between BAT distance embeddings
Azimuth cosine similarity Similarity between BAT direction embeddings

The default --mae-normalization mean averages the aligned time-frequency elements once and prints MAE multiplied by 100. Use --mae-normalization legacy only when reproducing the former extra frame-count division. Label the normalization in comparisons. All requested metric families must cover the same audio items; feature files use numeric arrays and are loaded without pickle.

Method and data

The paper introduces a Multimodal Pose Encoder and an Immersive Drama Transformer for binaural speech generation. The released MRSDrama data contains recorded drama, scripts, aligned annotations, video, poses, and prompts. See the paper for the generation architecture and model comparisons, and the evaluation guide for this repository's tools.

ISDrama architecture with the Multimodal Pose Encoder, Immersive Drama Transformer, and Drama-MOE.
Figure 4. ISDrama generation architecture described in the paper.

Citations

If you find this code useful in your research, please cite our work:

@article{zhang2025isdrama,
  title={ISDrama: Immersive Spatial Drama Generation through Multimodal Prompting},
  author={Zhang, Yu and Guo, Wenxiang and Pan, Changhao and Zhu, Zhiyuan and Jin, Tao and Zhao, Zhou},
  journal={arXiv preprint arXiv:2504.20630},
  year={2025}
}

Disclaimer

Any organization or individual is prohibited from using any technology mentioned in this paper to generate someone's speech without his/her consent, including but not limited to government leaders, political figures, and celebrities. If you do not comply with this item, you could be in violation of copyright laws.