AI voice agent latency: why sub-500ms matters to Australian callers

People judge a phone call by its rhythm before its content. When the pause after each sentence is short, callers relax and speak normally. When it's long, they start saying "hello?", talking over the agent or repeating themselves, and the transcript fills with noise. Latency, interruption handling and voice choice are the difference between a voice agent people tolerate and one they hang up on.

This is the technical-quality chapter of our guide to AI voice agents in Australia. It explains where delay comes from, why Australia adds its own challenges, and how to test before customers hear it.

Why half a second is the line to aim for

Human conversation runs on tight turn-taking. A study of ten languages published in PNAS found a universal tendency to avoid overlapping talk and minimise silence between turns, with differences between languages that were small in absolute terms. We're tuned to notice pauses that are longer than we expect, and on the phone, where there's no face to read, a long pause reads as confusion or a dropped line.

Telephone engineers have long planned around this too. The ITU's G.114 recommendation on one-way transmission time treats delay under about 150 milliseconds as essentially transparent for most applications and sets 400 milliseconds as a limit not to exceed in general network planning. That's network delay alone, before an AI has done any thinking. A voice agent has to fit listening, understanding, deciding and speaking inside a gap callers will accept.

That's why Fulcrum Lyre is engineered to a sub-500ms voice-to-voice latency target: measured from the end of the caller's speech to the start of the agent's audio. It's a target, not a promise for every call on every network, and you should measure it yourself.

Where the delay actually comes from

Two of these are distinctly Australian problems. Distance: if any stage runs in a data centre in the US or Europe, the round trip adds delay you can't optimise away. And tool calls: an agent that checks a calendar hosted overseas mid-sentence will pause, so look up availability before the caller asks where you can. Lyre runs on Australian infrastructure with a Sydney data-residency path and connects over Australian SIP (for example Symbio) or WebRTC, which keeps the network legs short. If data location matters to you for other reasons, see sovereign AI for Australian operators.

Interruptions matter as much as speed

Real callers interrupt. They say "yep, yep" while you're talking, correct a detail mid-sentence or cut in with "sorry, it's actually Thursday". An agent has to handle three cases differently:

• Back-channel noises ("mm", "yep", "right"): keep talking. Stopping every time sounds broken.
• Genuine interruptions ("no, wait"): stop within a fraction of a second, listen, and respond to the new information.
• Background noise (a dog, a radio, a grinder on site): ignore it rather than treating it as speech.

Get this wrong and you'll see it in the transcripts as repeated questions and "sorry, I didn't catch that". Tune it with real recordings from your callers' environments, not a quiet office.

Accents, voices and words that sound local

A voice that sounds imported can feel out of place on an Australian call, especially for a local business. Three things make an agent sound like it belongs:

The voice

Choose an Australian English (en-AU) voice and test it with people who resemble your callers. A voice that sounds fine to the team in the office may sound odd to an older caller in a regional town.

Understanding Australian speech

Recognition has to cope with broad and regional accents, fast speech, and the many Australians who speak English as a second language. Test with a range of speakers, not only native-born staff.

Vocabulary and formats

Suburb names (Toowoomba, Woolloomooloo, Mullumbimby), street types, mobile numbers spoken in groups ("oh-four-one-two"), dates in day-month order, prices in dollars including GST, and everyday words like arvo, servo and ute. If the agent mangles the caller's suburb, it has lost them. Give it your service-area suburb list and have it read addresses back.

If a meaningful share of your callers prefer another language, test that language separately before offering it. A poor experience in someone's first language is worse than a clear hand-off to a person who speaks it.

A pre-launch test plan

Run this before any customer hears the agent. Use real phones, not a browser demo.

1. Latency: make 20 calls from different mobile networks and record them. For each turn, measure the gap from the end of your speech to the start of the agent's reply. Note the median and the worst three.
2. Weak signal: call from a car, a lift lobby and a regional area with patchy coverage.
3. Noise: call from a worksite, a café and a moving car with the radio on.
4. Interruptions: cut in mid-sentence, correct a detail, and talk over the greeting.
5. Accents and speakers: have at least five different people call, including older and non-native English speakers.
6. Local data: give suburbs, street names, mobile numbers and dates and check the CRM entry matches.
7. Tool delays: ask for a booking and time the pause while the calendar is checked.
8. Hand-off: ask for a person and check how long a transfer or call-back takes.
9. Compliance lines: confirm the opening names the business, discloses the AI and recording, and that "stop" ends the call.

Score each call with the QA sheet in AI voice agent metrics, and don't launch until the worst calls are acceptable, not just the average ones.

What to watch once you're live

Latency regressions often come from changes nobody thinks of as voice changes: a longer prompt, a new tool call, a slower calendar integration. Watch greeting drop-off and "didn't understand" escalations every week; a sudden rise is usually a speed or audio problem before it's a script problem. For inbound deployments, the after-hours patterns in after-hours AI answering are a good place to start sampling, because those calls come from noisier places.

When latency isn't your biggest problem

A fast agent that says the wrong thing is still wrong. If your transcripts show wrong answers, missed escalations or pushy outbound behaviour, fix scope and content first. And if you're a developer comparing raw voice platforms on latency and control, our VAPI alternative comparison covers the trade-offs between developer APIs and an operator-ready setup.

For operators, the Fulcrum Lyre page explains how calls, SMS, CRM logging and channel recaps fit together; bring the test plan above and run it on your own call types.