Frame selection is the whole game: notes on making LLMs watch video

When using vision LLMs to process videos, selecting which frames to include is critical since models can only afford about 150 images per video—choosing the wrong frames means the model sees a slideshow rather than actually "watching" the content. The author describes a pipeline that uses adaptive scene detection (comparing frame scores against rolling averages rather than fixed thresholds) and multi-channel deduplication to intelligently select frames, ensuring every frame earns its place in the limited token budget rather than wasting space on near-identical shots or missing important moments.