The Documentary I Never Filmed

My attempt to automate a "documentary-style" video script generator using GPT-4o and a custom RAG pipeline completely collapsed during the final synthesis phase, and it took me three days of debugging to realize the issue wasn't my prompt, but how the model handled token-heavy context windows.

The goal was simple: feed the AI 50+ pages of interview transcripts and research notes, then have it output a structured script with timestamps and B-roll suggestions. For the first few chapters, it worked beautifully. But once the "context" hit about 15k tokens, the model started hallucinating events that never happened in the source text, or worse, it would just skip entire sections of the narrative.

The breaking point was this specific error I kept seeing in my logs when the script tried to parse the AI's output into my video editing software's XML format:

{
  "error": "Unexpected token '...' in JSON at position 4502",
  "status": 500,
  "message": "Malformed script structure: Missing closing brace for Scene 4"
}

At first, I thought the model was just "getting tired" or hitting a max output limit. I tried increasing the max_tokens parameter, but that did nothing. The AI was literally cutting off mid-sentence or replacing complex narrative arcs with ... [continue narrative here] ... because it was struggling to maintain the coherence of the long-form structure while adhering to the strict JSON schema I demanded.

I spent hours tweaking the system prompt, adding things like "Do not summarize" and "Be exhaustive," but it felt like shouting into a void. The diagnosis came when I stripped the RAG pipeline and fed the model the same data in smaller, 2k-token chunks. Suddenly, the hallucinations vanished and the JSON stayed valid.

The culprit was "lost in the middle" phenomenon. I was stuffing too much primary source material into a single prompt, and the model was prioritizing the beginning and end of the context window while treating the middle as optional filler. Because it was trying to maintain a "documentary tone," it filled those gaps with plausible-sounding lies to bridge the narrative.

I eventually solved it by implementing a Map-Reduce approach. Instead of one giant prompt, I wrote a script to:
Extract key narrative beats from each transcript chunk individually.
Synthesize a chronological outline based on those beats.
Generate the final script scene-by-scene, referencing only the specific beats for that segment.

Here is the prompt logic I ended up using for the "Map" phase to keep it grounded:

Act as a documentary researcher. Analyze the provided transcript segment. 
Extract only factual claims, direct quotes, and chronological events. 
Ignore any fluff or conversational filler. 
Output format: 
- Event: [Description]
- Quote: "[Exact text]"
- Timestamp: [MM:SS]

It's a frustrating lesson in AI architecture: just because a model can handle a 128k context window doesn't mean it should. The cognitive load of maintaining structural integrity (like JSON) while synthesizing massive amounts of data is where the wheels fall off. My "documentary" is still just a collection of text files, but at least the data is actually accurate now.