Multi-Voice Audiobooks: Solving Character Attribution

LeoMaker Expert 5h ago Updated Jul 25, 2026 99 views 7 likes 2 min read

Converting a novel into a multi-voice audiobook isn't actually a TTS problem; it's a parsing problem. The synthesis is the easy part—the real struggle is figuring out who is talking.

Why quotation marks aren't enough

A naive approach—simply finding text in quotes and assigning a voice—fails immediately when hitting real-world prose.

"I told you," she said, "and you did not listen."

In this case, one utterance is split across two quoted spans with the attribution wedged in the middle. A basic parser will split this into two separate lines and often assign them to different speakers.

The harder scenario is the total absence of tags:

"Are you coming?"
"In a minute."
"You said that an hour ago."

Humans track this alternation via context. A parser, however, has to model turn-taking and recognize exactly when that alternation breaks because a third character enters the scene.

Technical strategies that actually work

Lexicons over Regex: Using attribution verb lexicons (said, asked, whispered, muttered) and their inflections is far more reliable than raw regex. You need to explicitly handle both "..." said X and X said, "..." patterns rather than hoping a single expression catches everything.

The Coreference Necessity: "She said" is meaningless unless the system knows who "she" is in the current scene. Without a coreference resolution pass, you end up with "correct" attribution assigned to the wrong voice. This is actually worse than a monotone read because listeners immediately notice the inconsistency.

Alternation as a Prior: In untagged dialogue runs, alternating between the last two active speakers is usually correct. Treat this as a default that is overridden by named attribution, not as an absolute rule.

The document-level consistency constraint

One critical detail often overlooked: once a voice is assigned to a character, it must remain frozen. You can't have the protagonist's voice change in Chapter 14 just because the attribution confidence dipped.

This means voice assignment is a document-level decision. You cannot start rendering audio until the entire book has been parsed. For those building this incrementally, this architectural constraint is often a surprise.

A practical AI workflow for this looks like:

1. Full-text parse → create character inventory with mention counts.
2. Assign specific voices to the top N characters by frequency; assign a narrator voice to everything else.
3. Persist this mapping in a config file.
4. Render chapters against the frozen mapping.