colin mitts

video · multimodal

Video Intelligence

$ vidintel DcTlocbzf8W_01.mp4          # 0:31

› Finding scenes                        ✓ 17 scenes
› Transcribing speech                   ✓ 109 words
› Building the contact sheet            ✓ contact_sheet.jpg
› Choosing a lens                       ✓ Story · The voiceover is a personal
    morning-routine vignette praising cozy pajamas and coffee-making with
    her baby rather than teaching steps, so what happens matters more
    than what is said.
› Pass 1 · storyline from contact sheet ✓ 6 beats
› Pass 2 · captioning keyframes         ✓ 17 keyframes
› Writing the report                    ✓ report.html + report.pdf

7 model calls · 36,510 in / 2,087 out · ≈$0.21 · finished in 58s

a morning-routine reel by @kitschenista · 0:31 · 17 scenes · read as a story · 7 model calls · ≈$0.21 · 58s

two videos, one pipeline — it decides what kind of report to write. both reports untouched.

What it does

Give it a video, get back a report that explains it: a title and summary, the storyline broken into beats, and every scene with what was said and what the picture shows. It reads the video first to decide what kind of report to write — a recipe comes back as steps, a lecture as key points. Built solo, end to end — the version where I own every decision.

How it works 

video ──▶ ffmpeg scene cuts ──▶ one keyframe per scene
  └────▶ whisper ──▶ words, assigned to scenes by timestamp
                │
                ▼
contact sheet — the whole video in one look
(keyframe + time range + words spoken, per cell)
                │
                ▼
lens — decide what the video IS before describing it
(story · how-to · talk — and: is that even speech?)
                │
                ▼
pass 1  storyline: title · summary · beats · steps
pass 2  keyframe captions, storyline as context
                │
                ▼
report.html (scene player) + report.pdf

Every step caches and resumes; every run prints its token usage and cost. Structured outputs guarantee the shape of every answer — and what the schema can't express gets validated in code: beats are stitched to cover every scene exactly once, and every caption must name a real scene.

What broke + what I changed

Over a soundtrack, Whisper writes words nobody said — and they'd steer the whole report and fill it with fake quotes. The fix is a hallucination filter: segments are dropped when their words come back low-probability or loop a phrase until it compresses unnaturally well, with thresholds set against measured statistics from real clips. Lyrics transcribe cleanly, so the lens call catches those instead.

And the model can't be trusted to keep four near-identical frames in order. Every keyframe now carries its scene number burned into a banner, every caption must name the scene it describes, and responses are checked one-to-one — repeats dropped, skipped scenes re-asked on their own.