Part of AI Tools & Automation — automating real work with local AI models.
08_video_frame_benchmark.py · 17 sampled frames from a real 17-minute video · Qwen3.6-35B-A3B (multimodal, mmproj-F32) served locally · AMD PRO W7900 48 GB · Windows 11. The frame descriptions below are real model output.
A vision-language model cannot watch a video the way you do. It reads still images. So to make a VLM understand a video, you sample frames at intervals and describe each one. String those descriptions together and you get a timeline of what happened, without ever sending the video to the cloud.
This article builds that frame-sampling pipeline and measures it on a real 17-minute recording. It answers the two questions that matter: how fast is it, and does the result actually make sense?
- Sampling frames turns video understanding into a series of cheap image queries.
- The local VLM described each frame in 1.77 s on average, about 34 frames per minute.
- One frame per minute reconstructed the video’s full storyline accurately.
- The sampling interval is the key dial: more frames mean more detail and more time.
Why sample frames instead of watching video?
A minute of video at 30 fps is 1 800 frames. Feeding every frame to a VLM would be slow and pointless, because consecutive frames barely differ. Sampling solves both problems. You take one frame every few seconds, describe it, and skip the near-duplicates in between.
The interval sets the trade-off. A wide interval is fast but can miss brief events. A narrow interval catches everything but costs more time. For a talking-head or screen-recording video, one frame per minute already captures each scene change, as the benchmark below shows.
Pipeline architecture
The pipeline has three stages. ffmpeg extracts frames at a fixed rate, the VLM describes each frame, and a final step joins the descriptions into a timeline. Each stage is small and independent.
- Extract — ffmpeg samples one frame every N seconds and resizes it for the model.
- Describe — the local VLM answers “what happens in this frame?” per image.
- Assemble — the timestamped descriptions become a readable summary.
Implementation — frame extraction and description
ffmpeg does the sampling in one command. The fps=1/60 filter keeps one frame per minute, and the scale filter caps the width so the model spends tokens on content, not resolution.
# one frame per minute, 1024 px wide
ffmpeg -i video.mp4 -vf "fps=1/60,scale=1024:-1" frames/frame_%02d.jpgEach frame then goes to the VLM through the OpenAI-compatible vision API. Disabling thinking keeps the answer short and fast:
Free ebook
Free AI Video, Generated Locally
Working scripts and measured benchmarks. Free.
No spam. Unsubscribe at any time.
import base64
from openai import OpenAI
client = OpenAI(base_url="http://127.0.0.1:8082/v1", api_key="not-needed")
def describe(path: str) -> str:
b64 = base64.b64encode(open(path, "rb").read()).decode()
resp = client.chat.completions.create(
model="local-model",
messages=[{"role": "user", "content": [
{"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{b64}"}},
{"type": "text", "text": "Describe what happens in this video frame in one sentence."},
]}],
max_tokens=160, temperature=0.1,
extra_body={"chat_template_kwargs": {"enable_thinking": False}},
)
return resp.choices[0].message.content.strip()Benchmark methodology
The test extracted one frame per minute from a 17-minute video, giving 17 frames. Each frame was described once by the VLM, and the script recorded the processing time per frame. Seventeen frames give seventeen independent timing measurements of the same task, so the mean is stable.
Speed results
| Metric | Value |
|---|---|
| Frames analysed | 17 (one per minute) |
| Mean time per frame | 1.77 ± 0.34 s |
| Throughput | ~34 frames / minute |
| Total for the 17-min video | ~30 s |
The whole 17-minute video was summarised in about half a minute of compute. The first frame ran slightly slower because of the one-time warm-up, and every frame after settled near 1.7 s. At 34 frames per minute, even a dense sampling rate stays practical.
The reconstructed timeline — real output
Speed means nothing if the descriptions are wrong. They were not. Here is the actual, unedited timeline the model produced from the sampled frames:
| Time | Model’s description of the frame |
|---|---|
| 0:00 | A man with a beard and headset in a circular overlay while a software settings window is shown. |
| 2:00 | A user navigates the sidebar of an AI chat application, hovering over the model selector. |
| 4:00 | The presenter shows a slide titled “LOCAL AI”, illustrating how a laptop processes a question locally. |
| 6:00 | A user configures and loads the Gemma 4 model in a desktop application, with memory shown. |
Read top to bottom, the descriptions tell the video’s story: a presenter introduces local AI, opens a chat app, explains an offline architecture, then loads a model. The model even read on-screen text like the slide title and the model name. That is a genuine, searchable summary built entirely from still frames.
Choosing the sampling rate
The interval is the one dial worth tuning. For slow content — talks, tutorials, screen recordings — one frame per minute is plenty, and it kept the storyline intact here. For fast content — sports, action, rapid cuts — drop to one frame every few seconds so brief events are not missed.
Because cost scales linearly with frame count, the maths is simple. Halving the interval doubles the frames and roughly doubles the time. Start wide, then narrow the interval only around the moments you care about, which keeps the total cheap.
Sending multiple frames at once
For tighter context, you can send several frames in a single request and ask the model to describe the sequence. It then reasons about motion and change between frames, not just each still in isolation. This works well for short clips where continuity matters, though it uses more tokens per call.
content = []
for path in ["f1.jpg", "f2.jpg", "f3.jpg"]:
b64 = base64.b64encode(open(path, "rb").read()).decode()
content.append({"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{b64}"}})
content.append({"type": "text", "text": "These are consecutive frames. Describe what changes across them."})Detecting scene changes automatically
Because each frame has a text description, you can spot scene changes without any video-processing library. Compare consecutive descriptions, and when they diverge sharply, a new scene has begun. A cheap way to measure that divergence is word overlap: low overlap means the content shifted.
def scene_changed(prev: str, curr: str, threshold: float = 0.4) -> bool:
a, b = set(prev.lower().split()), set(curr.lower().split())
overlap = len(a & b) / max(1, len(a | b))
return overlap < threshold # low overlap = likely a new sceneRun that across the timeline and you get chapter markers for free. In the test video, the descriptions shifted clearly when the presenter moved from a slide to a live application, exactly where a human would draw a chapter boundary.
Building a searchable video index
The real payoff is search. Store each frame’s timestamp and description in a small database, and a video becomes queryable text. Ask “when does the Gemma model get loaded?” and a keyword or embedding search returns the exact minute.
index = [{"t": r["t_seconds"], "text": r["description"]} for r in frames]
hits = [r for r in index if "gemma" in r["text"].lower()]
# -> jump straight to the frames that mention GemmaScale that across a whole channel and you have a search engine over video content, built with nothing but ffmpeg, a local VLM, and a text index. No frames ever leave the machine, and the running cost is only electricity.
Use cases
- Video search: index every frame’s description so you can find the moment a topic appears.
- Content moderation: flag frames that match a policy without a human watching the whole video.
- Meeting and webinar notes: turn a screen recording into a timestamped outline.
- Highlight detection: spot scene changes by comparing consecutive descriptions.
Frequently asked questions
How many frames per minute can the model handle?
About 34 on this hardware, at 1.77 s per frame. That is fast enough to analyse a long video in seconds and to sample densely when a scene demands it.
Does one frame per minute lose important moments?
For slow content, rarely. It reconstructed this video’s full storyline. For fast cuts or brief events, narrow the interval so nothing slips between samples.
Can the model read text on screen?
Yes. It read slide titles and the on-screen model name directly from the frames, which makes it strong for tutorials and screen recordings.
Is it better to send frames one by one or together?
One by one is cheaper and scales linearly. Send several together only when you need the model to reason about change and motion across a short sequence.
Does the video ever leave my machine?
No. ffmpeg and the VLM both run locally, so the frames and the video stay on your own hardware from start to finish.
Summary
Frame sampling turns video understanding into a stream of quick image queries. On an AMD W7900, a local VLM described each frame in about 1.77 seconds — roughly 34 frames per minute — and one frame per minute was enough to reconstruct a 17-minute video’s storyline accurately, on-screen text included. The whole video was summarised in about half a minute of compute, entirely offline. Reproduce every number with the script: 08_video_frame_benchmark.py.
Companion code — every script and benchmark from this series lives in one repository: github.com/bestin-it/qwen-local-ai-scripts.
Free ebook
Free AI Video, Generated Locally
Run Wan 2.1 in ComfyUI on your own GPU — the scripts I use, measured times, sample clips. No cloud, no API keys.
No spam. Unsubscribe at any time.
Leave a Reply