What you are building
A video analyser is a pipeline, not a single model call. It takes a video in, extracts the audio, transcribes it to timestamped text, segments that text into chapters, pulls out the most important quotes, and produces a structured briefing you can read, search, or feed into other tools. Each stage is simple on its own; the craft is in passing clean, timestamped data from one stage to the next so the final output is accurate and traceable back to moments in the video. Build it on a short clip first, prove each stage works, then scale to longer content with chunking.
How it works
The pipeline has four stages. Transcription: extract the audio track (a tool like ffmpeg, or many transcription APIs accept video directly) and run it through Whisper — hosted via API for zero setup, or a local model if you have hardware — to get timestamped segments. Chunking into chapters: feed the transcript to an LLM and ask it to identify topic shifts and return chapter titles with start timestamps; for long videos, summarise in chunks first, then merge (a map-reduce approach) so you never exceed the context window. Quote extraction: instruct the model to return verbatim quotes from the transcript only, with their timestamps, and verify each quote is a real substring of the source. Briefing generation: assemble chapters, quotes, and an overall summary into a structured document — markdown or JSON — with every claim and quote linked to a timestamp.
Tips and pitfalls
The AI calls are the easy part; the plumbing causes the bugs. Keep timestamps attached to text at every stage so the final briefing can deep-link into the video and a human can correct chapter boundaries fast. When chunking long transcripts, overlap chunks slightly so a topic that straddles a boundary is not lost. For quotes, the rule “quote verbatim, never paraphrase, then I will verify” plus a substring check eliminates the subtle rewrites models tend to make. Use the smallest model that produces good chapters and summaries — this is summarisation, not frontier reasoning — and cache transcripts so reprocessing the analysis layer does not re-incur transcription cost. Start narrow, verify each stage, and the full pipeline comes together reliably.