shure-dev/small-vlm-sop-check
shure-dev/small-vlm-sop-check has 53 GitHub stars and 10 forks. GitHot measured 0 stars in 24 hours. Last checked 2026-11-16 03:09 UTC. Last push 2026-08-11.
Measured star growth
- in 24h
- 0
- in 7 days
- Update delayed
- in 15 days
- Update delayed
- in 1 month
- Update delayed
Net star changes over overlapping periods, not daily totals. Missing measurements are left blank. Stars measure attention, not software quality.
About
Catch procedure errors before they become incidents. Train and evaluate small VLMs for temporal grounding in first-person factory video.
Creative & MediaREADME.en.md
Originalindustrial-vlm-temporal-grounding
Toward a Connected Worker platform that understands factory procedures in real time and catches critical deviations before they cause harm.

Ten Factory Ego clips playing back to back. Orange shows human spans; blue shows Marlin-2B predictions.
Catch procedural mistakes before they become incidents
In a factory, a skipped step, incorrect order, unsafe tool operation, or missed inspection can lead to life-threatening incidents, equipment damage, quality defects, or production downtime. Reviewing recorded footage after the event cannot support the worker at the moment it matters.
The intended system continuously analyzes first-person video from a wearable camera with a small VLM. It should understand the current operation, completed steps, and steps that have not yet occurred, then notify a worker or supervisor when a critical deviation begins to emerge—before an incident or loss is final.
- Support work as it happens — frontline workers should be able to continue hands-free work while receiving an immediate indication of a skipped step or unsafe action
- Prevent incidents and loss — supervisors and manufacturing engineers should be able to respond before a deviation becomes an injury, defect, or equipment stop
- Run locally with low latency — models with at most 4B parameters are the primary target, allowing a future path to continuous inference on a wearable or nearby edge computer without sending sensitive footage to an external API
- Accumulate site-specific knowledge — precise event descriptions and spans can capture tools, parts, grips, and placements, while human corrections flow into fine-tuning and re-evaluation
As a foundation for that system, this repository focuses on temporal grounding: locating an event in time.
Input: video + "The worker turns a bag upside down and drops parts into a bin"
Output: the event's start/end timestamps, or absent
Real-time procedure analysis requires more than recognizing an object or action. The system must know when each operation starts and ends, in what order it occurs, and what has not occurred. Accurate timestamps can feed higher-level logic for the current step, omissions, order violations, and abnormal duration.
The current work first establishes this temporal capability on short clips. Temporal IoU (tIoU) measures the difference from human spans and provides one contract for improving event definitions, prompts, models, and training data. The purpose is to support safe and correct work while it is happening.
The source of advantage is not a larger general-purpose model. It is the ability to encode safety- and quality-critical site procedures as precise event definitions and timestamps, then continuously transfer that knowledge into a small model that can eventually run in real time.
Validating temporal understanding with Factory Ego
The primary pilot uses 20 fixed industrial first-person clips from Egocentric-10K. Each clip is 20 seconds at 2 fps. A human watches the footage, writes Japanese event descriptions, and marks the reference spans. External machine-generated annotations are not used as ground truth.
All 20 clips are now annotated with 68 events and 81 reference spans. We freeze 68 English queries that exactly match the current SOPs and compare candidate models on every 20-second clip using 2 fps, 640-pixel video and a shared timestamp-JSON contract. Times use half-open intervals from 0.0 seconds at the start of each clip.
| Model | Size | Predicted spans | mean tIoU | [email protected] P | R | F1 | Peak memory |
|---|---|---|---|---|---|---|---|
| Marlin-2B | 2B | 68 | 0.400 | 0.559 | 0.469 | 0.510 | — |
| Qwen3.5-4B | 4B | 68 | 0.178 | 0.206 | 0.173 | 0.188 | 5.64 GB |
| Qwen3.5-2B | 2B | 68 | 0.129 | 0.088 | 0.074 | 0.081 | 3.47 GB |
| Qwen3-VL-2B | 2B | 41 | 0.110 | 0.146 | 0.074 | 0.098 | 3.28 GB |
| Qwen3.5-0.8B | 0.8B | 64 | 0.076 | 0.078 | 0.062 | 0.069 | 2.30 GB |
| Gemma 4 E2B | E2B | 53 | 0.030 | 0.019 | 0.012 | 0.015 | 6.58 GB |
Marlin-2B uses its dedicated find(video, event) interface. The other models receive the same event text and JSON prompt one query at a time through 4-bit MLX conversions. Each model emits at most one span per event, which limits recall against the 81 references containing repeated actions. Every clip belongs to the development split, so these are diagnostic values rather than held-out test accuracy.
We also smoke-tested MiniCPM-V-4.6, InternVL3-2B, LFM2.5-VL-1.6B, three SmolVLM2 variants, two FastVLM variants, and Perception-LM-1B. MiniCPM returned a timestamp outside the 20-second contract; the others encountered compatibility errors in the current MLX conversion or video processor and therefore did not advance to full runs. Qwen2.5-VL-3B and Qwen3-VL-4B passed smoke tests but were explicitly excluded from this full matrix. Immutable runs, evaluation hashes, and failure reasons are linked from the aggregate artifact.
Improvement loop
flowchart LR
A[Operational video] --> B[Record events and spans<br/>by hand]
B --> C[Video VLM<br/>predicts spans]
C --> D[Compare on a timeline<br/>and with tIoU]
D --> E[Improve definitions,<br/>prompts, and models]
E --> C
D --> F[Export training data]
F --> E
One web app unifies annotation and results review:
- Thumbnail gallery — browse the 20 clips with per-clip progress and mean tIoU, sorted by lowest tIoU first
- Video-editor timeline — create and drag human spans next to model predictions on one screen; tIoU and F1 recompute live as spans move
- Dataset curation — clips excluded from training and evaluation are flagged in
datasets/<dataset>/curation.json
Translation, inference, and training remain reproducible CLI stages outside the app. The original Japanese annotation is human ground truth and is never overwritten by a model prediction.
Quick start
Python 3.10+ and ffmpeg are required. Factory Ego also requires accepting the access terms for builddotai/Egocentric-10K on Hugging Face.
python3 -m venv .venv
source .venv/bin/activate
python -m pip install -e ".[test,fetch]"
python tools/benchmark/fetch_factory_ego.py --apply
python tools/benchmark/validate.py --require-media
sop-app --dataset factory_ego
Annotation follows a video-first workflow:
- Watch the clip and decide which operational events need distinct labels.
- Describe the visible actor, object, and action precisely in Japanese.
- Add each occurrence at frame-level resolution.
- Use still frames and one-frame movement to refine boundaries.
- Review the full timeline for missing or overlapping events.
Edits autosave to datasets/<dataset>/annotations/human/<unit>.json. See the annotation guide for details.
Open the results in read-only mode with:
sop-view --dataset factory_ego
The writable sop-app is localhost-only. For network sharing, do not expose its unauthenticated editing API; use sop-view --host 0.0.0.0.
Two inference methods
| Method | VLM input | VLM output | Span construction | Role |
|---|---|---|---|---|
| Temporal Grounding | Video + event description | Start/end timestamps, or absent | Preserve the VLM span output | Primary |
| Frame Classification | One image at a time + question | Per-frame yes/no | Convert the answer sequence with rules | Baseline |
The primary method gives a video to a Video VLM and asks the model to produce temporal spans directly. The frame-classification rule engine is a comparison experiment for handling duration, short noise, and repeated occurrences in a yes/no sequence; it does not replace temporal-grounding output.
Both methods normalize to the same prediction format and use the same tIoU evaluation.
sop-check eval \
--ground-truth datasets/<dataset>/annotations/human/<unit>.json \
--prediction runs/<run-id>/predictions/<unit>.json
Bring your own video
sop-dataset init --dataset my_factory --name "My Factory"
sop-dataset add-video --dataset my_factory --unit clip_001 \
--video /path/to/private.mp4
sop-app --dataset my_factory
Events do not need to be predefined on the command line. Reviewers create them while watching the video. See the bring-your-own-data guide.
Training and data contract
Completed annotations can be exported with sop-export-ms-swift as video-SFT JSONL. LoRA/QLoRA runs use ms-swift as the external backend, and pre/post-tuning models are compared under the same contract and split.
datasets/ versioned metadata, event definitions, human GT, splits, hashes
data/ ignored videos, frames, audio, and preview media
runs/ immutable inference runs, raw output, normalized predictions
evaluations/ metrics locked to annotation and prediction hashes
training_runs/ training configuration and input locks; weights/logs ignored
Intervals use video-relative half-open seconds, [start_s, end_s). Human facts, model predictions, and evaluations remain separate. See the data contract.
Current scope
The current scope is offline annotation, prediction review, evaluation, and training export for short video clips. This stage builds the temporal model and ground truth required for real-time procedure analysis. Wearable deployment, streaming inference, alert logic, power consumption, and latency have not yet been validated.
The repository output alone is not intended to automate life- or equipment-critical safety decisions. Production deployment requires site-specific risk assessment, fail-safe controls, human oversight, and a clear boundary with existing safety systems.
Full videos, ordinary extracted frames, model weights, and personal data are not committed. The hero GIF is the only downsampled demonstration derivative containing Egocentric-10K imagery; attribution and terms are documented in docs/assets/README.md.
Validation
python -m pytest -q
python tools/benchmark/validate.py
sop-dataset validate --dataset factory_ego
python tools/quality/check_docs.py
python tools/quality/check_public.py
Code is released under the MIT License. External data, models, and checkpoints retain their own licenses and access terms.