Where sentence translation reaches its limits
Somewhere around minute four of a corporate training video, the machine translation quietly betrays you. The product name that was rendered one way in the opening line comes back wearing a different translation. The fitness instructor who has been chatting casually to camera suddenly sounds like she is reading a legal disclaimer. And the German line, because German expands, runs a full second longer than the audio slot it has to fit into.
None of these are translation errors. Every sentence, taken on its own, is perfectly fine.
That is precisely the problem.
Automatic dubbing is a cascade: speech is recognised, segmented, translated, and re-voiced — and every stage inherits the constraints of the one before it. There is, of course, another school: end-to-end pipelines that go further still, modifying the video itself so lips match the new language. These systems are improving fast, and for cinematic content they may well be the future. But for a large share of real production work: fitness videos, video games, training content, talking-head formats — what the audience judges is the audio: the voice, the tone, the timing. Lip sync is a nice-to-have; a wrong register or a broken product name is a defect. For those use cases, cascading remains the norm in production, and the quality of the cascade is decided stage by stage.
For years, the translation stage in pipelines like ours was handled by neural machine translation. NMT is fast, cheap, and remarkably good at what it was trained to do: turn a sentence into an equivalent sentence. But dubbing does not need an equivalent sentence. It needs a sentence that fits the scene, the speaker, the terminology, and the clock.
That gap is what motivated our latest R&D work: a dedicated translation service, exposed as an API, that replaces the NMT stage of our cascading dubbing pipeline with a controlled, LLM-based one.
Why NMT was the ceiling
A classic NMT engine is a sentence-in, sentence-out machine. It has no notion of who is speaking, what is on screen, what was said thirty seconds earlier, or how long the resulting line is allowed to be. Everything a dubbing director actually cares about, such as register, address form, terminological consistency, timing, sits outside the model's input. The model cannot get these things right or wrong; it simply cannot see them.
In practice, this blindness produced three recurring failure modes:
- Semantic drift across a scene. Each sentence was translated in isolation, so a term rendered one way at 00:01 could be rendered differently at 04:20, and the viewer notices long before the QA pass does.
- Register mismatch. A trainer speaking casually to camera and a corporate narrator received the same neutral, faintly bureaucratic output.
- Length overrun. Target languages that expand relative to the source produced lines that did not fit the available audio slot, pushing the problem downstream into manual post-editing.
You can mitigate each of these with glue code around an NMT engine: heuristics, post-processing, terminology injection tricks. We tried. You cannot solve them, because the model has no mechanism to act on the information. It is a ceiling built into the architecture.
What changed with LLMs
An LLM can be instructed. That single property, mundane as it sounds, is what makes it a better fit for the dubbing cascade. Not raw translation quality in the abstract, but the ability to condition the translation on everything the pipeline already knows and NMT had to ignore.
Our service turns that capability into an explicit, typed contract. A translation request no longer carries just a sentence; it carries the production context around it:
- Scene: a short description of the setting the line belongs to, so the model knows it is translating a fitness instructor speaking to camera, not a legal disclaimer.
- Context: the sentences already translated, so wording and terminology stay coherent across a whole asset instead of being re-decided sentence by sentence.
- Register: an explicit style parameter (informal, formal, technical, colloquial, marketing, and others), because tone is a production decision, not something a model should be left to infer.
- Address form: second person singular or plural, forced rather than guessed. In languages that grammaticalise formality and number, this is one of the most immediately visible quality signals to a viewer. Get it wrong and everything else stops mattering.
- Compression: a dubbing-specific instruction to produce a deliberately shorter rendering while preserving meaning, tone, and terminology. The length problem, addressed inside the translation step rather than repaired after it.
Notice the pattern: each of these is a parameter the pipeline sets. Nothing depends on the model happening to guess the production intent. The translation step stops being a black box you hope behaves and becomes an instrument the pipeline plays.
Terminology: keeping the deterministic part deterministic
There is a temptation, once you have an LLM in the loop, to hand it everything, including the customer glossary, and hope. We resisted it, for a simple reason: generative models are the right tool for semantics and style, and the wrong tool for "this client's product is always called X." A rule is not something to be creative about.
So the service includes a dedicated linguistic matching layer, built on spaCy, that scans each sentence against the customer glossary before any generation happens. It works on lemmas rather than surface forms, so it recognises a term through inflection, plurals, hyphenation variants, casing, compounds, and Unicode quirks, and returns the exact character offsets of every hit.
Glossaries are compiled once into an indexed structure and cached, so matching against thousands of terms adds negligible latency per sentence.
Only the terms actually found in the sentence are then injected into the model instruction, with a directive to inflect them as the target grammar requires. The result is a hybrid: deterministic where determinism matters, generative where fluency matters. Ambiguous terms are surfaced with their alternatives rather than silently resolved; a human decision stays a human decision.
The matching layer has also proven useful on its own: the API exposes it as a separate endpoint, with no LLM call involved.
Engineering notes
The service itself is deliberately boring which, in production infrastructure, is a compliment. It is a small FastAPI application with a narrow surface: match, translate, or both in a single call.
Language support is a configuration entry, not a code change. Glossaries can be pre-compiled ahead of a job so the first sentence of a batch is not slower than the rest, and subtitle files can be uploaded directly for end-to-end testing.
Correctness is held in place by a regression suite that runs real-world sentence variations against a full production-scale glossary, because a terminology guarantee you don't test is just a terminology hope.
What this unlocks
The shift from NMT to a controlled LLM stage is not a swap of one black box for another. It is a change in what the translation step is allowed to know. Semantics survive across a scene. Register is a parameter rather than an accident. Terminology is enforced rather than encouraged. Length is negotiated during translation instead of repaired after it.
For a cascading dubbing pipeline, that means fewer decisions pushed downstream, and less human post-editing standing between the translated script and the voice that finally speaks it.
The fitness instructor at minute four sounds like herself, calls the product by its name, and finishes her line exactly when the audio slot ends.
Which is all a viewer ever asked for.
If you want to see the rest of the story, Voiseed Studio is where it's told.
