Latency budgets for real-time voice agents

On a phone call, silence is loud. A pause of half a second feels like the other person is thinking. A pause of a second and a half feels like the line has dropped. People fill the gap with "Halo? Unanisikia?" and then talk over whatever the system finally says.

That is the bar for Waridi Agents. A caller in Mwanza with a weak signal should get a reply that feels as quick as a person at a counter would give. To get there, we treat latency as a budget. Every stage of a turn gets a slice of time, and every slice has an owner.

What a single turn looks like

A turn starts when the caller stops speaking and ends when they hear the first sound of the agent's reply. In between, a lot happens:

1. Endpointing. We decide the caller has finished, not just paused.
2. Final transcription. Waridi Sikia settles the last words of the utterance.
3. Response. The language model reads the conversation and starts producing a reply.
4. Speech synthesis. Waridi Sauti turns the first words of that reply into audio.
5. Transport. Audio travels back over the network and the phone line to the caller.

Users never see these stages. They only feel the sum.

The budget

Here is the budget we design against for a single turn. These are targets, not measured results. We publish them so the people building with us know what we are aiming for and can hold us to it.

Turn latency targets (caller stops speaking -> first audio heard)

  endpointing decision        150-250 ms
  final ASR transcript         50-80 ms   (most text already streamed)
  LLM first token             120-200 ms
  TTS first audio chunk        60-100 ms
  network + telephony legs     80-170 ms
  ---------------------------------------
  target total               about 500 ms, always under 800 ms

The ranges are wide on purpose. A call over a clean fibre link in Dar es Salaam and a call relayed through a congested rural tower are different problems. We would rather show honest ranges than one flattering number.

Endpointing is the biggest lever

The single largest piece of the budget is deciding when someone has finished talking. A plain voice activity detector waits for a fixed stretch of silence. Set it too short and the agent interrupts people mid-thought. Set it too long and every turn drags.

Swahili makes this harder in an interesting way. Speakers often trail off with "eeh", "sasa", or a rising "...na?" while they look for the next word. Those are not the end of a turn. Neither is a long breath before a phone number. So our endpointing model looks at more than silence. It also considers the words Sikia has already heard and the shape of the pitch. A sentence like "Nataka kulipa bili ya..." is clearly unfinished, so the agent waits. "Asante, hiyo tu." is clearly done, so the agent can move.

Stream everything

The budget only works if no stage waits for the previous one to finish completely.

• Streaming ASR. Sikia sends partial transcripts every few hundred milliseconds while the caller is still speaking. By the time endpointing fires, most of the text is already in hand. The final pass only confirms the last word or two.
• Early preparation. We can start preparing likely context, such as account lookups and retrieval, on stable partials before the turn ends.
• Token-level handoff. The model's output is fed to Sauti as it arrives. We cut at natural phrase boundaries so the first spoken chunk sounds complete, not clipped.
• Chunked audio out. Sauti returns the first audio frames before the full sentence is synthesised. The caller hears the reply begin while the rest is still being generated.

Barge-in

Real people interrupt. A caller hears "Salio lako ni shilingi elfu..." and cuts in with "Hapana, nataka kuongea na mtu." The agent has to stop talking at once, throw away the rest of its planned reply, and listen.

That means barge-in detection runs during playback, with echo cancellation, so the agent does not mistake its own voice for the caller's. Our target is to stop playback within about 200 ms of the caller starting to speak. Anything slower and the caller ends up shouting over a machine.

Backchannels and fillers, used carefully

Sometimes a turn genuinely needs more time, like a balance check against a slow backend. Dead air is the worst outcome here. A short, natural acknowledgement is better: "Sawa, ngoja kidogo nikuangalie." It tells the caller they were heard and buys a second or two honestly.

We are careful with this. Fillers should be rare, short and truthful. An agent that says "hmm" before every answer sounds nervous, not human. We only use them when a stage is expected to go over budget, never to hide a slow system.

Where the servers are matters

Physics sets a floor under the budget. A round trip from East Africa to a data centre in Europe or North America can eat a large share of the transport slice before any model has run. For real-time voice, that is too much.

So regional inference is part of the design, not an optimisation for later. We are building toward running Sikia, the language model and Sauti close to the callers they serve. We also want to keep the telephony edge, SIP or WebRTC, in the same region, so audio does not cross a continent twice in one turn. Jitter buffers on phone lines add their own delay too. We size them for real network conditions rather than lab ones.

How we hold ourselves to it

Every turn in our test calls is logged with a timestamp at each stage boundary. That lets us see which stage blew the budget, not just that the turn was slow. We look at the slow tail, the worst calls of the day, as much as the median. The caller who waits two seconds remembers it far longer than the one who waited half a second.

If you are building on Waridi Agents, the streaming docs and the telephony guide explain how to keep your own part of the pipeline inside the budget. Early access is open at get started.