Multimodal Research Hub - Vision-Language Models (VLMs)

A living resource for Vision-Language Models & multimodal learning
(papers, models, datasets, benchmarks, tools, ethical challenges, research directions)

🔗 Seminal models (Post-2021)

2025

MiniCPM-o 2.6 - An 8B parameter multimodal model capable of understanding and generating content across vision, speech, and language modalities
Janus-Pro-7B - autoregressive framework that unifies multimodal understanding and generation
Gemma3-4b-it - One of the smallest multimodal models with a 128k token context window, supporting 140+ languages
Qwen2.5-VL-3B-Instruct - capable of handling various tasks ranging from localization to document understanding and agentic tasks, with a context length of up to 32k tokens
GLM-4.6V - GLM-4.6V series model includes two versions: GLM-4.6V (106B), a foundation model designed for cloud and high-performance cluster scenarios, and GLM-4.6V-Flash (9B), a lightweight model optimized for local deployment and low-latency applications
MiniCPM-V - On-device friendly (<6 GB RAM), scores 70% on MMBench, supports 30+ languages & 1.8 M-pixel images
CogVLM2-Video - Temporal-aware, based on CogVLM2, adds 3D RoPE for long video clips, reaches 63% on Video-MME
Ovis - Decoder-only, native dynamic tiling (AnyRes++), 128 k context, SOTA on OCRBench & DocVQA
MobileCLIP2 - Image-text pair encoder designed as drop-in vision tower for FastVLM-style pipelines, 2× faster than CLIP-L at same zero-shot top-1
FastVLM - Uses a hybrid vision encoder "FastViTHD" to reduce token count and encoding latency for high-resolution images, achieves up to 85× faster Time-to-First-Token (TTFT) relative to LLaVA-OneVision, with a vision encoder that is ~3.4× smaller
PerceptionLM (PLM) - Focuses on detailed image & video understanding, includes “Perception Encoder (PE)” variants that outperform SigLIP2 (image) and InternVideo2 (video) in benchmarks
DeepSeek-VL2 - MoE, 3B/16B params (3B active). Efficient for field IoT, scientific diagnostics, outperforms Cambrian-1 on MMMU at half the latency
Eve - Efficient multimodal model with “Elastic Visual Experts,”, 1.8B parameters, How to bridge the performance gap between encoder-free and encoder-based VLMs?
Kwai Keye-VL - Decoder-only, 8B params, Qwen-3-8B backbone. Tailored for short-form vertical-video understanding & mobile-first VQA. Released by Kuaishou, Apache-2.0
Qwen2.5-Omni - End-to-end 7B any-to-any model (audio-image-video-text). Shares Qwen2-Audio encoder + Qwen2.5-VL ViT, with unified tokenizer for all modalities, Apache-2.0
OmniGen2 - Hybrid decoder + VAE. Trained on LLaVA-OneVision & SAM-LLaVA mixes, supports any-to-any generation
BLIP3-o - Decoder-only, 4B & 8B variants. Fine-tuned on 60K GPT-4o-generated instruction-image pairs, excels at open-ended captioning and creative editing
InternVL3 - Follows the "ViT-MLP-LLM", integrating InternViT with various pre-trained LLMs. Comprises seven models ranging from 1B to 78B parameters. Supports text, images, and videos simultaneously. Features Variable Visual Position Encoding (V2PE) for better long-context understanding. Utilizes pixel unshuffling and dynamic resolution strategy for efficient image processing. Training data includes tool usage, GUI operations, scientific diagrams, etc. Can be deployed as an OpenAI-compatible API via LMDeploy's api_server
Falcon-2 11B VLM - Decoder-only, CLIP ViT-L/14 dynamic encoder, Apache-2.0. Built for multilingual OCR & fine-grained visual grounding
Eagle 2.5 - Decoder-only architecture with 8B parameters, designed for long-context multimodal learning, featuring a context window that can be dynamically adjusted based on input length. Utilizes progressive post-training to gradually increase the model's context window. Achieves 72.4% on Video-MME with 512 input frames, matching the results of top-tier commercial models like GPT-4o and large-scale open-source models like Qwen2.5-VL-72B and InternVL2.5-78B
Llama 3.2 Vision - Decoder-only architecture with 11B and 90B parameters, supports 8 languages - English, German, Italian, Portuguese, Hindi, Spanish, and Thai. NOTE: for image+text applications, English is the only language supported. Features a 128k-token context window, optimized for multimodal complex reasoning
VLM-R1 - Based on DeepSeek's R1 method, combined with Qwen2.5-VL, excels in Representational Expression Comprehension (REC), such as locating specific targets in images. Uses RL and SFT to enhance performance. Supports joint image and text processing. Provides open-source training code and dataset support
Kimi-VL - An efficient open-source MoE based VLM. Activates only 2.8B parameters in its language decoder (Kimi-VL-A3B). Demonstrates strong performance in multi-turn agent interaction tasks and diverse vision language tasks. Features a 128K extended context window for processing long inputs. Utilizes native-resolution vision encoder (MoonViT) for ultra-high-resolution visual inputs. Introduces Kimi-VL-Thinking variant for strong long-horizon reasoning capabilities
Molmo Series - Four models by Ai2: MolmoE-1B, Molmo-7B-O, Molmo-7B-D, and Molmo-72B. MolmoE-1B based on OLMoE-1B-7B, Molmo-7B-O on OLMo-7B-1024, Molmo-7B-D on Qwen2 7B, and Molmo-72B on Qwen2 72B. Known for high performance in vision-language tasks, with Molmo-72B outperforming many proprietary systems
Skywork-R1V - Open-sourced multimodal reasoning model with visual chain-of-thought capabilities. Supports mathematical and scientific analysis, and cross-modal understanding. Released quantized version (AWQ) for efficient inference
DeepSeek Janus Series - Unified transformer architecture with decoupled visual encoding pathway, available in Janus-Pro (7B and 1B), Janus (1.3B), and JanusFlow (1.3B) parameter sizes, supports English and Chinese languages, designed to handle sequence lengths of up to 4,096 tokens
Pixtral 12B & Pixtral Large - Decoder-only architecture with 12B (Pixtral 12B) and 124B (Pixtral Large) parameters, combines a 12B multimodal decoder with a 400M vision encoder. Pixtral Large integrates a 1B visual encoder with Mistral Large 2. Supports dozens of languages and features a 128k-token context window. Developed by Mistral AI, known for its efficiency in handling interleaved image and text data
Gemma3 - Decoder-only architecture with 1B/4B/12B/27B parameters, supporting multimodality and 140+ languages, featuring a 128k-token context window and function calling capabilities, based on Google's Gemini 2.0 architecture
PH4 Mini - Decoder-only architecture with specialized parameters, designed for efficient completion tasks, using optimized transformer layers and Microsoft's Phi-4 framework as pretrained backbone
Aya Vision 32B - Decoder-only architecture with 32B parameters, designed for advanced visual language understanding, supporting 23 languages and featuring dynamic image processing capabilities
Ola - Decoder-only architecture with 7B parameters, using OryxViT for vision encoder and Qwen-2.5-7B, SigLIP-400M, Whisper-V3-Large, BEATs-AS2M(cpt2) as pretrained backbone
Qwen2.5-VL - Decoder-only architecture with 3B/7B/72B parameters, using redesigned ViT for vision encoder and Qwen2.5 as pretrained backbone
Ocean-OCR - Decoder-only architecture with 3B parameters, using NaViT for vision encoder and pretrained from scratch
SmolVLM - Decoder-only architecture with 250M & 500M parameters, using SigLIP for vision encoder and SmolLM as pretrained backbone

2024

2023

2022 & Prior

📊 Datasets

🏆 Benchmarks

GEOBench-VLM (2025) - For Geospatial Tasks
EgoExoBench (2025) - Designed to evaluate cross-view video understanding in MLLMs, contains paired egocentric–exocentric videos and over 7,300 MCQs across 11 subtasks
AV-Reasoner (2025) - Improving and Benchmarking Clue-Grounded Audio-Visual Counting for MLLMs
EOC-Bench (2025) - Can MLLMs Identify, Recall, and Forecast Objects in an Egocentric World?
RBench-V (2025) - A Primary Assessment for Visual Reasoning Models with Multi-modal Outputs
LENS (2025) - A multi-level benchmark explicitly designed to assess MLLMs across three hierarchical tiers—Perception, Understanding, and Reasoning—encompassing 8 tasks and 12 real-world scenarios.
On Path to Multimodal Generalist: General-Level and General-Bench (2025) - Does higher performance across tasks indicate a stronger capability of MLLM, and closer to AGI?
VisuLogic (2025) - Benchmark for Evaluating Visual Reasoning in Multi-modal Large Language Models
V2R-Bench (2025) - Holistically Evaluating LVLM Robustness to Fundamental Visual Variations
Open VLM Leaderboard HF(2025)
SVLTA (2025) - Benchmarking Vision-Language temporal alignment via synthetic video situation
Fewshot-VLM (2025) - Evaluates adaptation to new tasks with limited examples
OCRBench & OCRBench v2 (2024) - OCR capabilities of Large Multimodal Models
MMMU-Pro - contains 28k college-level questions delivered as screenshots with varying fonts/layouts
MMT-Bench (2024) - comprises 31,325 curated multi-choice visual questions from various multimodal scenarios such as vehicle driving and embodied navigation, covering 32 core meta-tasks and 162 subtasks in multimodal understanding
ZeroShot-VLM (2023) - Tests generalization to unseen tasks without task-specific training
MetaPrompt (2023) - Measures domain generalization with unseen prompts/domains
MMBench (2023) - Multimodal benchmark
MMMU (2023) - Massive Multi-discipline Multimodal Understanding benchmark
ChartQA (2022) - Question Answering about charts with visual and logical reasoning

Video-Language Benchmarks

RTV-Bench (2025) - Real-time video analysis, contains 552 videos (167.2 hours) and 4,631 QA pairs
SciVideoBench (2025) - Benchmark for scientific video reasoning, covering disciplines in Physics, Chemistry, Biology, and Medicine
Video-MME v2 (2025) - Updated 900 → 1800 videos (508 h), adds audio-only & subtitle-only splits, now the default long-video benchmark for GPT-5 & Gemini-2.5-Pro
MT-Video-Bench (2025) - Video Understanding Benchmark for Evaluating Multimodal LLMs in Multi-Turn Dialogues
Video-MMLU (2025) - A Massive Multi-Discipline Lecture Understanding Benchmark
VLM²-Bench (2024) - Evaluates multi-image/video linking capabilities (9 subtasks, 3K+ test cases)
ViLCo-Bench (2024) - Continual learning for video-text tasks

Dynamic Evaluation

VLM4D (2025) - 4-D spatiotemporal reasoning benchmark
OmniSpatial (2025) - Towards Comprehensive Spatial Reasoning Benchmark for Vision Language Models
SPBench (2025) - Spatial Perception and Reasoning Benchmark
MMSI-Bench (2025) - A Benchmark for Multi-Image Spatial Intelligence
Spatial-DISE (2025) - Evaluating Spatial Reasoning in Vision-Language Models
LiveXiv (2024) - Monthly-changing benchmark to prevent overfitting, estimates true model capabilities
MM-Vet (2023) - Evaluating Large Multimodal Models for Integrated Capabilities

Specialized Tasks

ScienceQA (2022) - 21K science questions for multimodal reasoning
OK-VQA (2021) - 11K open-ended questions requiring external knowledge

🔍 Research Directions

Regression Tasks

VLMs often struggle with numerical reasoning, counting, measurement tasks.
Hybrid numeric modules, symbolic-differentiable integration, specialized heads should be suggested.

Diverse Visual Data

Expansion to non-RGB modalities - multispectral, depth, LiDAR, thermal, medical imaging.
Dealing with domain shift, modality gap, alignment strategies.

Multimodal Output Beyond Text

Generation of images, videos, 3D scenes (like with diffusion + VLM couplings)
Dense prediction tasks - segmentation, layout generation, vision + language fused output
Challenge - coherence across modalities, consistency, speed

Multitemporal & Continual Learning

Lifelong / online VLMs that adapt over time
Avoid catastrophic forgetting across visual domains
Enable adaptation to evolving visual environments

Efficient Edge / Deployment

Quantization, pruning, distillation, adapters, LoRA for vision-language models.
Mobile / ARM / GPU runtimes, optimized kernels.
Example - FastVLM is a major step in efficient inference design.

Multimodal Alignment & Fusion

Methods - gated cross-attention, mixture-of-prompts, dynamic modality weighting, feature modulation.
LaVi (vision-modulated layernorm) would be a good example
Address density mismatch - dense visual features vs discrete text tokens

Embodied AI / Vision-Language-Action (VLA)

VLMs extended to output robot actions given vision + instructions
Classic example- RT-2 (Google DeepMind) as early vision–language–action foundation
LiteVLP - lightweight, memory-based vision-language policy for robot tasks, showing improvements over existing baselines.
Challenges - sample efficiency, latency, generalization, sim-to-real

Temporal Reasoning

Video question answering, event prediction, cross-frame consistency.
Need memory modules, temporal fusion, temporal attention.

Medical & Domain-Specific VLMs

Domain shift, hallucination, safety-critical constraints.
Combine domain-specific modules, calibration, prompt tuning.
Require robust uncertainty estimation, out-of-distribution detection.

⚠️ Ethical Challenges

Bias Type	Prevalence	High-Risk Domains	Mitigation Effectiveness
Gender	23%	Career images	63% reduction (Counterfactual)
Racial	18%	Beauty standards	58% (Adversarial)
Cultural	29%	Religious symbols	41% (Data Filtering)
Hallucination	34%	Medical reports	71% (CHAIR metric)
Spatial Reasoning	High	Scene understanding	Requires further research
Counting	Moderate	Object detection	Requires specialized techniques
Attribute Recognition	Moderate	Detailed descriptions	Needs improved mechanisms
Prompt Ignoring	Moderate	Task-specific prompts	Requires better understanding of intent

Visual privacy & identity leakage - reconstructing faces or sensitive information from embeddings
Adversarial / backdoor attacks in visuals - e.g. adversarial patches
Copyright / image usage / dataset licensing
Geographic / demographic bias in visual datasets
Multimodal RLHF / preference bias - mixed visual + language feedback
Explainability in multimodal decisions
Misuse / dual-use risk - surveillance, deepfakes, misinformation
Mitigation strategies - adversarial robustness, counterfactual data, filtering, human oversight

🔒 Privacy Protection Framework

graph TD
    A[Raw Data] --> B{Federated Learning?}
    B -->|Yes| C[Differential Privacy]
    C --> D[Secure Training]
    B -->|No| E[Reject]

VLMs often process sensitive data (medical images, personal photos, etc.). This framework prevents data leakage while maintaining utility:

→ Federated Learning Check

Purpose: Train models on decentralized devices without raw data collection
Benefit: Processes user photos/text locally (e.g., mobile camera roll analysis)
Why Required: 34% of web-scraped training data contains private info (LAION audit)

→ Differential Privacy (DP)

# DP-SGD Implementation for Medical VLMs
optimizer = DPAdam(
   noise_multiplier=1.3,  
   l2_norm_clip=0.7,      
   num_microbatches=32
)

Guarantees formal privacy (ε=3.8, δ=1e-5)
Prevents memorization of training images/text

→ Secure Training

Homomorphic Encryption: Process encrypted chest X-rays/patient notes
Trusted Execution Environments: Isolate retinal scan analysis
Prevents: Model inversion attacks that reconstruct training images

→ Reject Pathway

Triggered for:
- Web data without consent (23% of WebLI dataset rejected)
- Protected health information (HIPAA compliance)
- Biometric data under GDPR

↓ Real-World Impact

Scenario	Without Framework	With Framework
Medical VLM Training	12% patient ID leakage	0.03% leakage risk
Social Media Photos	Memorizes user faces	Anonymous embeddings
Autonomous Vehicles	License plate storage	Local processing only

🛠️ Tools

VLMEvalKit - Unified evaluation framework supporting 20+ benchmarks (Apache-2.0)
VLM-BiasCheck - Bias auditing toolkit covering 15+ dimensions (MIT)
Prompt tuning / adapter frameworks - CoOp/CoCoOp, VLSM-Adapter, FedMVP, CHROME

Optimization Toolkit

4-bit QAT - Quantization-aware training via bitsandbytes (3.2× speedup)
Flash Attention - Memory-efficient attention with xFormers (2.8× speedup)
Layer Dropping - Structural pruning via torch.prune (1.9× speedup)

📌Emerging Applications

Robotics / embodied agents (vision + language + action)
AR / XR / smart glasses - real-time vision + language understanding overlays
Document / UI understanding - combining layout, OCR, semantics
Scientific / diagram reasoning - charts, formulas, visuals + language
Satellite / geospatial + textual metadata fusion
Medical imaging + clinical note fusion
Assistive technologies - interactive image description + QA for visually impaired
Visual-to-code / UI generation - sketches or mockups → code
Video summarization / captioning / QA over long videos

🤝Contributing

Thank you for considering contributing to this repository! The goal is to create a comprehensive, community-driven resource for Multimodal and VLM researchers. Contributions ranging from updates to models, datasets, and benchmarks, to new code examples, ethical discussions, and research insights would be welcomed:)

Measured star growth

About

README.md

Multimodal Research Hub - Vision-Language Models (VLMs)

Table of contents