Turn detection
Voice activity detection and an end-of-speech pause decide when the user is done. Too short and the agent interrupts; too long and it feels slow. We tune it per use case and language.
AI Agents · Voice
Voice agents that answer in time.
And stop when you interrupt.
A voice agent chains speech-to-text, a language model with tools, and text-to-speech, all streaming at once. Users judge it in the first second: how long it waits, whether it talks over them, and whether it mispronounces a name. We engineer the whole chain, from the microphone to the voice, and measure it on real audio in your languages.
A specialist practice of MLAIA Data ScienceLed by Dr. Yochai Edlitz · Ph.D., Weizmann Institute
The pipeline
Three models in a row.
Each one adds delay.
Every stage of a voice pipeline adds time: deciding the user has finished speaking, finalising the transcript, the model's first tokens, and the first audio from the voice. We stream every stage and start speaking at the first complete sentence, instead of waiting for the whole reply.
Streaming speech-to-text
ElevenLabs Scribe, Amazon Transcribe, Deepgram, Google, Azure or Whisper-family models, chosen by language, accuracy on your vocabulary, and streaming delay. Low-confidence fragments are dropped instead of answered.
The agent
A language model with tools, such as booking, lookup or record updates, released one sentence at a time. Exact values like dates and amounts can be shown on screen rather than spoken from free text.
Text-to-speech
ElevenLabs, Google Chirp and WaveNet, Amazon Polly, Azure or Cartesia voices, compared on naturalness, pronunciation and time to first audio. A fallback voice takes over if the primary one is slow.
Barge-in
When the user starts talking, the agent pauses at once. A cough or an "mm" resumes playback; a real new request stops the turn. Echo cancellation keeps the agent from hearing itself.
Speech-to-speech models
Models such as Amazon Nova Sonic or OpenAI's realtime models listen and speak in one step, which can cut delay. We compare them with a pipeline on your languages and tool needs before choosing.
What we have built
A Hebrew and English voice assistant,
with an avatar that speaks.
For a healthcare app, currently in development, we built a voice assistant with a lip-synced avatar. It uses Amazon Transcribe streaming for Hebrew speech recognition in an EU region, Claude on Amazon Bedrock for the agent, and sentence-by-sentence speech so the avatar starts talking early. If a cloud voice responds too slowly, the sentence falls back to the device's own voice.
Hands-free mode includes soft barge-in: speech while the assistant is talking pauses it, and only a relevant final transcript interrupts the turn. We ran comparisons of Hebrew and English voices across several providers before choosing defaults, and kept demo voices off sensitive data unless the provider's contract covers it. Voice work builds on our audio and acoustics practice, including noise reduction and echo cancellation.
We describe client projects in general terms. Details are shared in conversation, subject to confidentiality.
Hebrew & multilingual voice
Hebrew needs its own tests.
Not a translated English setup.
Hebrew speech recognition and synthesis behave differently from English. Missing vowels make names and rare words ambiguous for a voice, gendered forms of address matter, and many callers switch between Hebrew and English in one sentence.
Pronunciation
Adding vowel marks or phonetic hints for names and terms before synthesis, and testing the voice on your vocabulary, not on sample sentences.
Recognition on your terms
Custom vocabularies and evaluation on recordings that include product names, addresses and code-switching.
Gender & register
Addressing the user in the right grammatical gender and register, which affects both the model's text and the chosen voice.
Voice stack optionsElevenLabs ScribeAmazon TranscribeDeepgramGoogle SpeechWhisperElevenLabsGoogle Chirp & WaveNetAmazon PollyCartesiaNova SonicWebSocket & WebRTCLiveKitAmazon ConnectTwilio
A practical delivery path
Measure on real audio.
Then tune the slowest stage.
Define the conversation
Who calls, about what, in which languages and on which channel, and which actions the agent may take.
Bake-off
Compare speech-to-text and voices on your recordings and vocabulary, with delay measured per stage.
Build & tune
Streaming pipeline, turn detection, barge-in and fallbacks, tuned against recorded test conversations.
Operate
Monitor delay, interruptions, transcription confidence and handoffs, with privacy rules for what is logged.
Before we start
Good questions. Straight answers.
How fast can a voice agent respond?
It depends on the channel, models and network. We measure each stage on your setup and stream all of them, so the agent starts speaking at its first sentence instead of after the full reply.
Can we use ElevenLabs?
Yes, ElevenLabs is one of the voices we integrate and compare. For personal or health data, we check the provider's data terms first and use a provider with the right contract where needed.
Can a voice agent work over the phone?
Yes, through Amazon Connect, Twilio or a SIP provider. Phone audio is narrower-band and noisier than web audio, so recognition and turn detection are tested on real calls.
What happens when the agent cannot help?
It says so and hands off: transfer to a person, a callback request, or a message with a summary of the conversation so far, depending on the channel.
Do you record the calls?
Only if you decide to and the user is informed. Many of our designs keep no audio or transcripts at all and log only which tools were used.
Keep exploring
Related reading & tools.
AI Agents & Voice Agents · Voice Agents
Tell us what
you’re working on.
Share the problem, the data you have and what success would look like. We’ll discuss a practical next step.
Please don’t include confidential datasets, credentials or patient information.