Voice & Speech

The Voice & Speech card, on the editor's Conversation step, controls how the agent sounds and how it shares the conversation. A voice is required; the rest have defaults that work.

The card itself holds Allow barge-in, Words before interrupting, the voice cascade and Caller language recognition. The slider button in the card's header opens a Speech Tuning panel with the rest: Turn detection, a Thinking cue, Voice activity detection, and the selected provider's own voice character settings.

The whole card is Voice only. It is absent, and so is the requirement for a voice, on an agent with Voice switched off in its Channels.

Allow barge-in

Barge-in decides what happens when the caller starts talking while the agent is talking. It's on by default, which means caller speech interrupts the agent mid-sentence — the agent stops, and listens.

Leave it on for nearly everything. Callers interrupt naturally, and an agent that talks over them feels broken.

Turn it off only when the agent must be heard to the end: a required disclosure, a recording notice, a legal script. With barge-in off, the caller can't stop the agent mid-utterance, so keep those utterances short.

Words before interrupting

Below the switch is Words before interrupting — how much the caller has to say before their speech actually stops the agent. Leave it empty and any sound interrupts, which is what every agent does today: a cough, a throat-clear, or an "mhm" of agreement cuts the agent off mid-sentence. Set it to 3 and none of those three stops the agent, because none of them is three words.

Two things that aren't obvious:

• 1 is not the same as empty. Empty uses speech detection; 1 still switches to transcribed words, so it adds the delay without filtering anything. 2 is the smallest setting that actually filters a one-word backchannel.
• It only applies while the agent is talking. An agent waiting for the caller still takes a single word as the start of a turn, so a one-word answer to a question is never swallowed.

It refines barge-in rather than replacing it, so it does nothing at all while Allow barge-in is off — an agent playing a disclaimer to the end has no interruption for a word count to refine. The field stays visible and editable there anyway, so the value is still yours to change. 1–10.

Choosing a voice

The voice is chosen through four dependent dropdowns, each one populated by the choice above it. Pick them in order:

Changing any level clears the levels below it, so switching language means re-picking the voice and engine, because a voice from one language isn't valid in another.

Some models produce speech themselves rather than handing text to a separate voice service. When you select one of those on the Brain card, the provider is fixed by the model and the dropdown is replaced with a note saying so.

Multiple languages

Multiple languages is enabled per organization. If the Voice & Speech card has no Languages section, it isn't on for yours yet. If it is turned off after an agent was given more languages, the rows stay as saved and calls use the primary language only. You can remove a row there (its greeting parts go with it), but you can't add or change one.

One agent can hold a call in up to five languages: a primary voice and up to four more, each with its own voice. Calls start in the primary language. When the caller speaks another of the languages, or asks for it, the agent switches its voice and replies in that language.

Add a language

In the Languages section, the voice you chose above becomes the Primary language. Select Add Language to add a row, then fill it in like the primary voice: provider, language, voice, and engine. Rows can use different providers. A row is titled with its language, voice, and provider, and carries an Incomplete badge until all four are picked. The delete button on a row removes it.

Rules the editor enforces:

• Five languages in all, the primary plus four. At the limit Add Language is disabled, with the reason beside it.
• One voice per language family. en-US and en-GB can't both be on one agent, because the agent knows both as English and switches by language name. The picker hides a dialect of a language already listed. If a draft got there another way, Save fails with <language> is the same language as <language>. Use one voice per language family.
• A language needs a speech recognition peer. A language that Amazon Transcribe can't transcribe while streaming is disabled in the picker with "no speech recognition". If the primary voice's language is one of these, Add Language is disabled. See Recognition language for which languages these are.
• Not available with a model that speaks in its own voice, because that model takes no extra voices.

Removing a language that a greeting part is written in is refused until you change or remove that part. See Greeting.

How the agent switches

The caller is transcribed in the language the call is in. Amazon Transcribe listens for that one language, so speech in another of the agent's languages reaches the agent as the nearest-sounding words in the current one: "Können Sie bitte Deutsch sprechen?" can arrive as "Kenzie Bitterdeutschprechen." The agent is told to read the caller's words for what they sound like, and switches the call in one of two ways:

• It replies in another of its languages. The reply is spoken in that language's voice, and the call continues in it (see Which voice speaks). The language a reply starts in is the one that counts: a reply that repeats a line in a second language speaks that line in the second language's voice and keeps the call where it was.
• It calls the built-in switch_language tool when the caller asks for another language or names one in answer to a menu. See Switch language.

From then on the caller is transcribed in the new language. The agent is told not to switch on a single name or filler word in another language.

A caller who starts speaking another language without asking can be misheard on that first turn, because it is transcribed in the current language. A menu in the greeting makes the switch dependable, for example "For Spanish, say español."

The call's language follows the call. It carries across a transfer to another agent, a return from a workflow, and a reconnect after a dropped call.

Which voice speaks

The agent's voice always follows the language of what the agent says. Each reply is spoken in the voice for the language the reply is written in. If the agent replies in English, the English voice reads it, even when the caller was last heard in German. If the reply is in a language the agent has no voice for, the voice stays as it was. A reply in another configured language also becomes the call's language, so the next turn continues in it unless the caller switches again.

The greeting and the thinking cue

A multi-language agent can play its greeting in several voices, one part per language. See Greeting. A returning-caller greeting you wrote yourself is one block of text, so it's spoken in a single voice: the one for the language the call is in when it starts. The thinking cue works the same way: the text you write is spoken in the voice of the language the call is currently in, and isn't translated.

If a voice fails

Each language keeps its own failover. An ElevenLabs voice that fails falls back to an Amazon Polly voice for the same language.

If the provider's key can't be used when a call starts, a language that has a Polly voice (Spanish, for example) is spoken by that Polly voice from the first word.

Some languages have no Polly voice: Croatian, Greek, Filipino, Hindi, Indonesian, Malay, Slovak, Ukrainian, and Vietnamese. A language from that list, as an additional language on a voice from another provider, is left out of the call when the provider's key can't be used as the call starts. The call continues in the languages that remain. The primary voice is never left out: if its language is one of these, an English voice speaks it. If the key is accepted at the start and fails during the call, an English voice reads that language aloud. The language's row in the Languages section shows a note when this applies.

Custom vocabulary

Tandem builds a vocabulary for each language's speech recognition when you publish. Calls use it only once every language's vocabulary is ready. Until then calls run without recognition tuning, in every language. The prompt part of vocabulary isn't affected.

Recognition language

Next to the voice cascade is Caller language recognition — a separate setting from the voice's own language. Leave it blank and it follows the voice's language automatically, so a Spanish-language voice transcribes a Spanish-speaking caller correctly with no extra setup.

Override it for the asymmetric case: a caller who speaks one language to an agent that answers in another — for example, a caller speaking Spanish to an agent whose voice, and therefore replies, are in English. Without an override, that call would be transcribed as English and the agent would consistently mishear the caller.

The dropdown offers every Transcribe streaming language, which is a narrower set than the languages Polly can speak — the list is the whole supported set, so anything absent from it has no streaming counterpart. See Troubleshooting for what happens when a voice's language is one of those.

With more than one language, the dropdown is replaced by a list of the agent's languages, and you can't override it. The caller is transcribed in whichever of them the call is in. Adding the first extra language clears an override you had set, and a notice on the card says what it was. Removing the last extra language brings the dropdown back, set to follow the voice.

Voice character

Once a voice is chosen, the Advanced disclosure adds provider-specific tuning for how that voice sounds. Every field is optional — leave it blank and the vendor's own default applies, so a config that predates these fields keeps behaving exactly as it did before.

Polly voice character

Shown when the selected voice provider is Polly.

• Speaking rate — a prosody keyword (x-slow, slow, medium, fast, x-fast) or a percentage from 20% to 200% inclusive. Anything else is rejected when you save. For example, to slow an agent down, set it to 90%.
• Volume — a prosody keyword (silent, x-soft, soft, medium, loud, x-loud) or a decibel offset such as +2dB or -6dB. The offset's format is checked when you save, but its range isn't — an out-of-range value is only caught by Polly itself, when it fails to synthesize speech, not when you save the agent.

ElevenLabs voice character

Shown when the selected voice provider is ElevenLabs.

• Speed — 0.7 to 1.2.
• Stability — 0 to 1.
• Similarity boost — 0 to 1.
• Style exaggeration — 0 to 1.
• Speaker boost — on or off.
• Text normalization — auto, on, or off. This is ElevenLabs' own handling of digit runs, currency, and dates in the text it speaks (for example, expanding $5 to "five dollars"), not a setting Tandem applies itself.

On the v3 Conversational and v4 Turbo engines only Stability applies. The other settings are saved but ElevenLabs ignores them for those models.

Pronunciation dictionaries

Pronunciation dictionaries sits under the ElevenLabs voice character settings. It lists the pronunciation dictionaries in your organization's ElevenLabs workspace, read with the key on your ElevenLabs integration, and applies the ones you pick to every line the agent speaks. Use one for a product name, a brand, a drug name, or a surname the voice gets wrong. For example, a dictionary with the rule GLaDOS → ˈɡlædoʊz makes the agent say the name that way every time.

• You can pick up to 3. Once 3 are picked, the rest of the list is disabled until you remove one.
• Each call uses a dictionary's latest version. An edit you make to a dictionary in ElevenLabs reaches calls within five minutes, with no change to the agent.
• You create and edit dictionaries in ElevenLabs. Tandem lists them and applies them; it doesn't edit them.
• If your ElevenLabs integration isn't connected, the field says so and lists nothing. Connect it, then use the refresh button beside the field.

A dictionary is not checked when you save. Each call checks it with ElevenLabs instead. If ElevenLabs reports it doesn't exist (it was deleted, or it lives in a different ElevenLabs workspace from the connected key), that call goes ahead without it. If ElevenLabs doesn't answer in time, the call can't get the latest version, so it goes ahead without that dictionary too. Once the list of your dictionaries has loaded, one still saved on the agent but no longer in your workspace shows as (not found in ElevenLabs), so you can remove it. While the list is loading, fails to load, or ElevenLabs isn't connected, a saved dictionary shows as its id alone.

Dictionaries don't apply everywhere:

• v3 Conversational and v4 Turbo don't support pronunciation dictionaries. The field says so, and the saved dictionaries are ignored on those engines.
• If ElevenLabs fails during a call and the agent switches to its Amazon Polly standby voice, Polly doesn't apply the dictionaries.

Through the API, the field is slots.tts.settings.elevenlabs.pronunciationDictionaries in the agent configuration, and List pronunciation dictionaries returns the ids. An entry is { "id": "…" }, or { "id": "…", "versionId": "…" } to pin one version instead of following the latest. A pinned dictionary is still checked at call time and skipped if ElevenLabs reports it doesn't exist. If ElevenLabs doesn't answer in time, a pinned dictionary is used anyway, since the call already has its version. The pinned version itself can't be checked (ElevenLabs has no way to list a dictionary's versions), so pin only a version you know exists.

Thinking cue

When the agent calls a tool — looking up an account, running a workflow — there's a pause while that work happens. By default the line is silent, and callers take a silent line for a dropped call.

The Thinking cue is a short phrase spoken while a tool call runs:

One moment while I check that…

Keep it generic. It plays before any tool call, so it can't reference a specific lookup. Keep it short, too. It should fill a gap, not become the pause. On a new agent, leaving the field empty means no cue at all, which is fine for agents with no tools.

Turn detection

Turn detection is how the agent decides the caller has finished — the other half of barge-in, which decides when they've started. There are two strategies, chosen under Strategy.

Silence timers is what every agent has always used, and it stays the default. The agent waits for the caller to stop making sound, then waits a little longer in case they resume. It's cheap and completely predictable, and it can't tell a thinking pause from a finished sentence, because all it measures is silence. Everything under Voice activity detection below tunes it.

Smart turn model runs a small speech model that predicts whether the caller is done from how they sounded — falling intonation, a phrase that resolved — rather than from silence alone. A caller who says "so my account number is… hold on, let me find it" gets waited for instead of answered into.

When smart is worth it

Reach for it when callers are being cut off mid-sentence and raising End-of-turn grace has made every other reply feel sluggish. Silence timers cannot separate those two cases: one number sets both the tolerance for a pause and the delay on every turn. A model separates them.

Stay on silence timers when turn-taking is already fine, when calls are short and transactional ("say or enter your account number"), or when you need turn-taking to behave identically on every call — a model makes a prediction, and predictions vary.

What stops applying in smart mode

The rest of the VAD fields still do their jobs: Silence before reply, Detection threshold, Speech start delay, and Minimum volume are about detecting sound, not about deciding a turn is over. What changes is that the silence timer now only marks where the caller stopped speaking — the model, not the clock, decides whether that was the end.

Smart-mode tuning

Choosing the smart strategy reveals three more fields, each optional — leave one blank and the model's own default applies. Once any of them holds a value it stays on screen on silence timers too, so you can see and clear what you set; it just isn't read there. Only smart mode reads them.

Silence limit — the longest silence the model will sit through before calling the turn over regardless of what it predicted. A backstop on a mispredicted turn, not the normal wait. 100–10,000 ms; default 3,000.

Pre-speech padding — how much audio from just before the caller started speaking the model gets to hear. The start of a phrase is part of what it reads. 0–5,000 ms; default 500.

Audio window — the longest stretch of caller audio the model scores at once; anything older is trimmed off the front. 1–30, default 8. Note the unit: this field is in seconds where the other two are milliseconds, and fractions are allowed.

Change one at a time and test on a real call, the same as the VAD fields.

Voice activity detection

Voice activity detection (VAD) is how the agent hears that the caller is talking, and — on the silence-timer strategy — how it decides they've finished a turn. Five fields tune it, each individually optional: leave any of them blank and the underlying voice pipeline's own default applies.

Silence before reply — how long the caller must stop talking before the agent counts the turn as over. Milliseconds; default 200.

End-of-turn grace — after Silence before reply elapses, a further window in which the caller may resume talking without the agent replying. Milliseconds; default 600.

The two add together. What a caller experiences as "how long until the agent replies" is Silence before reply + End-of-turn grace, not either one alone. With both at their defaults, that's 0.2s + 0.6s = 0.8s total. Raising or lowering only Silence before reply and expecting turn-taking to feel different is why this used to feel untunable — the grace window kept adding its own 0.6s regardless of what Silence before reply was set to.

End-of-turn grace applies to the silence-timer strategy only. With Turn detection set to the smart turn model, the model decides end-of-turn and this field is never read — the card says so under the field, and Silence limit is not a replacement for it. See What stops applying in smart mode.

Detection threshold — how confident the detector must be that the sound it hears is speech. Raise it on noisy lines so background noise doesn't register as talking; lower it for soft speakers who get treated as silence. 0–1; default 0.7.

Speech start delay — how long incoming speech must continue before it counts as the caller starting a turn, rather than a stray sound. Milliseconds, 0–2,000; default 200.

Minimum volume — the quietest sound level still treated as speech rather than background noise. 0–1; default 0.6.

Leave all five empty unless you have a specific problem to solve. When you do change them, change one at a time and test on a real call — several of them interact, and moving more than one at once makes it impossible to tell which one helped. Clearing a field and saving now restores the pipeline's own default for it; it no longer leaves the old value in place.

Limits and defaults

Troubleshooting

The agent talks over the caller. Confirm Allow barge-in is on. If it is, check Words before interrupting — a threshold set there means short utterances no longer stop the agent at all. If that's empty too, the caller's speech isn't registering: lower the VAD Detection threshold so quieter speech counts.

The agent interrupts itself, or stops for background noise. The detector is treating noise as speech. Raise the Detection threshold.

The agent still stops for a cough or an "mhm". Set Words before interrupting to 2 or 3. Accept that genuine interruptions get a little slower in exchange — see the warning under that field.

Interrupting the agent now takes a beat too long. Lower Words before interrupting, or clear it. Any value at all, including 1, switches interruption detection over to transcribed words; only an empty field uses plain speech detection.

The agent cuts callers off mid-sentence. Raise the VAD Silence before reply so it waits longer before deciding the turn is over.

The agent cuts in when the caller pauses mid-sentence to think. Raise End-of-turn grace — it's the window that lets the caller resume without the agent jumping in. Remember it delays every reply by the same amount, so raise it only as far as your callers need. If you've reached the point where every reply feels slow, that's what the smart turn model under Turn detection is for.

I switched to the smart turn model and my tuning stopped mattering. End-of-turn grace isn't read in that mode — the model decides end-of-turn instead. Silence limit is a backstop on a mispredicted turn, not the same knob. Switch back to silence timers and your saved values apply again, unchanged.

There's an awkward gap before every reply. Lower Silence before reply or End-of-turn grace — together they set the total wait. If the gap only happens on some replies, it's a tool call rather than turn-taking — set a Thinking cue.

The line goes silent for several seconds mid-call. That's a tool call with no cue. Set a Thinking cue.

A custom audio thinking cue turned into a spoken phrase. Someone typed text into Thinking cue and saved, which overwrites an audio-file cue set through the API. Re-apply the audio cue through the API, and leave the box empty on future saves from this card.

I cleared the Thinking cue and it's gone, but I wanted to keep the audio-file cue. Clearing and saving now removes whatever cue was set, spoken or audio-file — see the warning above. Re-apply the audio cue through the API if you cleared it by mistake.

The voice sounds robotic. Try a higher Voice Engine tier for the same voice, if the voice offers one.

Save fails asking for a voice. The cascade is incomplete. All four levels must be filled. Re-pick from the top, since changing one level clears the ones below it.

The agent speaks correctly but the transcript reads like a different language, or the agent keeps misunderstanding a caller who's speaking the voice's own language. The voice's language may have no Transcribe streaming counterpart, so the caller is being transcribed as English regardless of what the voice speaks. This is a known gap for Bulgarian, Hungarian, Tamil, Turkish, Welsh, and Icelandic — the six voice languages with no streaming peer. Arabic is supported in both variants: Modern Standard Arabic and Gulf Arabic each transcribe in their own variant. The Caller language recognition dropdown is the definitive list: a language that isn't in it is a language callers can't be transcribed in. There's no indicator on the voice cascade itself today, so if the agent's voice is one of these languages, set Caller language recognition explicitly to the closest supported language you expect callers to use, or choose a different voice language.

The agent didn't switch when the caller started speaking another language. The caller is transcribed in the call's current language, so a short phrase in another language can read as unrelated words, and the agent is told not to switch on a single word. Offer a menu in the greeting ("For Spanish, say español") so the agent can switch on the caller's answer with switch_language.

A language is missing from the language picker. Either it has no Transcribe streaming language, or another voice on the agent already covers its language family (es-ES and es-US are both Spanish).

Spanish callers are recognized as the wrong regional Spanish. Recognition follows the voice. An ElevenLabs Spanish voice (es) is recognized as Spanish from Spain (es-ES), and an Amazon Polly es-US voice as Spanish from the United States (es-US). Pick the voice whose region matches your callers.

Calls run without my vocabulary in one language. Calls use custom vocabulary only once every language's vocabulary is ready, so one language still building, or failed, holds back all of them. The status line on the Vocabulary card shows the first language that isn't ready.

Next steps