My LLM Pipeline Debugging Log: Naming & Rendering

Misnamed functions are a silent killer in AI workflows. We just dealt with a "Sonnet-canary leak" where a function called resolve_local_model was actually returning a paid model pin. This caused our satellite phases—like self-consistency rails—to bill at cloud rates instead of staying local. The fix was simple but necessary: rename everything to be explicit (resolve_writer_model vs resolve_local_writer_model) so the code doesn't lie to the dev.

Beyond the naming, we hit two major rendering issues in our video pipeline:

1. Silent Shot Drops: Long-form renders were shorter than planned because the system just skipped failed shots. I implemented a fallback ladder in shot_list_renderer.py using a _backfill_pass. If the primary source fails, it tries a substitute before hitting a branded card.
2. Frozen Tails: In 9:16 shorts, visuals and narration were sized independently. When the script ran long, the compositor just froze the last frame. We solved this by adding narration_fit to proportionally rescale scenes to match the actual TTS duration.

On the data engineering side, we found a nasty quirk with B2 storage. Running restic forget --prune doesn't actually reclaim billed space unless you have a version-expiry lifecycle rule active. I've updated the runbook and improved the offsite_backup_failed alerts to include the actual stderr for faster debugging.

To stop guessing about token usage, we added deep observability to our two-pass writer path. We're now tracking prompt-size metrics specifically for:

RAG snippets

Research data

Dev diary bundles

Internal grounding anchors

Now that we have this telemetry on the dashboard, we can actually start trimming the bloat in our prompt engineering.

For those building similar LLM agents, here is the logic we used for the render-fit fix:

def render_shot_list(scenes, tts_durations):
    # Prevent "frozen tails" by rescaling scene durations 
    # to match actual TTS output
    adjusted_scenes = _fit_scene_durations(scenes, tts_durations)
    return adjusted_scenes

The gap between the architect's plan and the actual render is finally closing.