Real-time Photorealistic Dynamic Scene Representation and Rendering with 4D Gaussian Splatting

Project page | Paper

Real-time Photorealistic Dynamic Scene Representation and Rendering with 4D Gaussian Splatting,
Zeyu Yang, Hongye Yang, Zijie Pan, Li Zhang
Fudan University
ICLR 2024

4D Gaussian Splatting: Modeling Dynamic Scenes with Native 4D Primitives,
Zeyu Yang, Zijie Pan, Xiatian Zhu, Li Zhang, Jianfeng Feng, Yu-Gang Jiang, Philip H.S. Torr
Fudan University, University of Surrey, University of Oxford
Arxiv preprint

This repository is the official implementation of "Real-time Photorealistic Dynamic Scene Representation and Rendering with 4D Gaussian Splatting". In this paper, we propose coherent integrated modeling of the space and time dimensions for dynamic scenes by formulating unbiased 4D Gaussian primitives along with a dedicated rendering pipeline.

🛠️ Pipeline

Get started

Environment

The hardware and software requirements are the same as those of the 3D Gaussian Splatting, which this code is built upon. To setup the environment, please run the following command:

git clone https://github.com/LogosIntelligenceLab/4d-gaussian-splatting
cd 4d-gaussian-splatting
conda env create --file environment.yml
conda activate 4dgs

Data preparation

DyNeRF dataset:

Download the Neural 3D Video dataset and extract each scene to data/N3V. After that, preprocess the raw video by executing:

python scripts/n3v2blender.py data/N3V/$scene_name

DNeRF dataset:

The dataset can be downloaded from drive or dropbox. Then, unzip each scene into data/dnerf.

Running

After the installation and data preparation, you can train the model by running:

python train.py --config $config_path

🎥 Videos

🎞️ Demo

🎞️ Dynamic novel view synthesis

https://github.com/LogosIntelligenceLab/4d-gaussian-splatting/assets/45744267/5e163b88-4f70-4157-b9f5-8431b13c26b7

🎞️ Bullet time

https://github.com/LogosIntelligenceLab/4d-gaussian-splatting/assets/45744267/ac5bc3b2-dd17-446d-9ee6-6efcc871eb84

🎞️ Free view synthesis from a teleporting camera

https://github.com/LogosIntelligenceLab/4d-gaussian-splatting/assets/45744267/6bd0b57b-4857-4722-9851-61250a2521ab

🎞️ Monocular dynamic scene reconstruction

https://github.com/LogosIntelligenceLab/4d-gaussian-splatting/assets/45744267/2c79974c-1867-4ce6-848b-5d31679b6067

📜 BibTex

@inproceedings{yang2023gs4d,
  title={Real-time Photorealistic Dynamic Scene Representation and Rendering with 4D Gaussian Splatting},
  author={Yang, Zeyu and Yang, Hongye and Pan, Zijie and Zhang, Li},
  booktitle={International Conference on Learning Representations (ICLR)},
  year={2024}
}

@article{yang20244dgs,
    title={4D Gaussian Splatting: Modeling Dynamic Scenes with Native 4D Primitives},
    author={Yang, Zeyu and Pan, Zijie and Zhu, Xiatian and Zhang, Li and Feng, Jianfeng and Jiang, Yu-Gang and Torr, Philip HS},
    journal={arXiv preprint},
    year={2024},
}