Skip to content
Projects

ixlab/vidformer has 71 GitHub stars and 6 forks. GitHot measured 0 stars in 24 hours. Last checked 2026-11-16 22:26 UTC. Last push 2026-11-10.

How these counts are measured

Measured star growth

in 24h
0
in 7 days
Collecting history
in 15 days
Collecting history
in 1 month
Collecting history

Net star changes over overlapping periods, not daily totals. Missing measurements are left blank. Stars measure attention, not software quality.

About

A drop-in declarative optimization framework for rendering video data visualizations.

Creative & Media

README.md

Original

vidformer

Paper arXiv DOI Test PyPI version Crates.io Version Open In Colab License

Vidformer brings instant playback to data-edited videos, enabling rapid exploration of multimodal datasets. See our ACM MM '26 paper, Vidformer: Enabling Instant Playback for Interactive Video Exploration, or the preprint on arXiv, for details.

Developed by the OSU Interactive Data Systems Lab.

Vidformer teaser

Quick Start

To quickly try out Vidformer you can:

Or, you can deploy it yourself:

git clone https://github.com/ixlab/vidformer
cd vidformer
docker build -t igni -f Dockerfile .
docker-compose -f vidformer-igni/docker-compose-local.yaml up

You can find details on this in our Getting Started Guide.

Why vidformer

Exploring multimodal datasets means editing videos with data: "show me every time X happens", "draw a box over each Y", "overlay Z telemetry on this video". Scripts are the only tool expressive enough for these tasks, but they take minutes or hours to render. Vidformer gives scripts the responsiveness of a video editor: just swap import cv2 with import vidformer.cv2 as cv2 and play the result instantly.

Vidformer combines three techniques:

  • Declarative Lifting: An API shim runs your existing code with symbolic frames, recording a declarative specification of each output frame instead of rendering it.

  • Optimized Rendering: With the full specification known in advance, the rendering engine transparently optimizes and parallelizes rendering, just like a relational database optimizes relational queries.

  • Video on Demand: Result videos are served as a standard VOD stream, with each segment rendered just in time, so playback latency is independent of video length.

Vidformer usually renders videos 2-3x faster than cv2, and starts playback in 0.25-0.5 seconds (~400x faster) when serving videos on demand. When operated by an LLM agent, Vidformer enables conversational video exploration: ask for a video in natural language and watch the result in seconds.

Vidformer builds on open technologies you may already use:

  • OpenCV: A cv2-compatible interface ensures both you (and LLMs) can use existing knowledge and code.
  • Supervision: Supervision-compatible annotators make visualizing computer vision models trivial.
  • FFmpeg: Built on the same libraries, codecs, and formats that run the world.
  • Jupyter: View transformed videos instantly right in your notebook.
  • HTTP Live Streaming (HLS): Serve transformed videos over a network directly into any media player.
  • Apache OpenDAL: Access source videos no matter where they are stored.

Documentation

About the project

Cite:

@inproceedings{winecki2026_vidformer,
  title     = {Vidformer: Enabling Instant Playback for Interactive Video Exploration},
  author    = {Winecki, Dominik and Nandi, Arnab},
  booktitle = {Proceedings of the 34th ACM International Conference on Multimedia},
  series    = {MM '26},
  year      = {2026},
  location  = {Rio de Janeiro, Brazil},
  publisher = {Association for Computing Machinery},
  address   = {New York, NY, USA},
  isbn      = {979-8-4007-2213-4},
  doi       = {10.1145/3767308.3836399},
  url       = {https://doi.org/10.1145/3767308.3836399},
}

An earlier version is available as a preprint on arXiv.

File Layout:

Vidformer components are detailed here.

License: Vidformer is open source under Apache-2.0. Contributions are welcome.

Acknowledgements: Supported by the Imageomics Institute (NSF Award #2118240) and NSF Award #1910356. Any opinions, findings and conclusions or recommendations expressed in the materials here are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.