The industry has spent a lot of energy making voice agents sound human. I understand why: natural pacing, interruption handling, and expressive speech are immediately impressive. But after building voice systems that interact with real scheduling and operational tools, I think fluency has become an overly convenient definition of quality.
A voice agent becomes a real product the moment it can do something outside the conversation. Answering a question is useful. Verifying an account, checking an available time, changing a reservation, or writing a structured note is a different class of responsibility. The model is no longer producing only language; it is coordinating state in another system while a person waits on the phone.
That changes what I look for. The hard part is not whether a model can select a function in a clean demonstration. The hard part is maintaining one coherent turn while audio, transcripts, model output, tool arguments, downstream services, and the caller’s next interruption all arrive on different clocks. A warm voice that confidently confirms an action before it actually completes is worse than a slightly mechanical system that tells the truth.
I have found it useful to treat tool use in voice as a protocol, not a prompt trick. The protocol has explicit states, deadlines, and evidence for what completed. Natural language still matters, but it sits on top of a small state machine that refuses to confuse a partial stream with a finished decision.
One turn, several clocks
A typical turn can involve at least five timelines:
- Speech recognition decides whether the caller has finished.
- The model streams words or a tool request.
- Text-to-speech begins playing a response.
- A tool performs work against an external service.
- The caller interrupts, corrects, or asks a new question.
If these timelines are allowed to coordinate implicitly, strange failures appear. The agent may begin speaking a confirmation before a tool commits. A half-streamed tool request may look parseable even though the model never finalized it. A slow service may finish after the conversation has already moved on. An interruption may cancel audio but leave an operation running without a clear owner.
The fix is not one enormous timeout. Each boundary needs its own meaning. I separate the budget for waiting on a finalized model decision, waiting for speech to become safe, executing the external operation, and preserving a small outer margin for cleanup. Those budgets should be centralized so the outer pipeline cannot expire before the inner stages have had the time the product promises them.
This is less glamorous than model selection, but it is what makes the system explainable. When a turn fails, the event should say which contract expired: model finalization, speaking coordination, tool execution, or overall turn budget.
Final means final
Streaming APIs create a subtle temptation: execute as soon as an object can be parsed. That is unsafe. Parseability and finality are different facts.
Consider a tool with no arguments. An empty argument buffer can legitimately mean {} after the model has finalized the tool request. The same empty buffer can also mean that the stream stopped early, the connection dropped, or the model had not emitted arguments yet. Treating both cases as executable turns an incomplete response into a real side effect.
My rule is strict: a tool request must have an explicit completion signal before it can enter the execution queue. Argument validation happens after that gate. A completed zero-argument request may normalize to an empty object. A partial request is discarded even if a parser could manufacture a valid value from it.
That rule also helps with replay. Reconnection and retry logic can inspect stable identifiers and finalized events instead of guessing from fragments. The system can say, “this tool decision completed but execution did not,” which is much safer than replaying every fragment seen before a disconnect.
Separate conversation from execution
The model should express intent, but a coordinator should own execution. I like a narrow coordinator with four responsibilities:
- accept only finalized, schema-valid requests;
- serialize or reject conflicting operations;
- attach deadlines and cancellation behavior;
- return a durable result envelope to the conversation.
The result envelope should distinguish success, expected business rejection, temporary dependency failure, and unknown outcome. Those are different conversational situations. “That time is no longer available” is not the same as “the scheduling service did not respond.” The first invites another choice. The second should avoid pretending the requested change did or did not happen.
Idempotency belongs here too. If an operation can be retried, the coordinator needs a stable request identity or a downstream idempotency key. If it cannot be made safely repeatable, the correct recovery path is to verify current state before trying again.
Speaking is part of correctness
Text systems can wait silently. Voice systems cannot. A pause that feels normal at 400 milliseconds feels broken at several seconds. The obvious response is to fill the space with language, but careless filler creates another failure mode: the agent can talk over the result or imply that work is complete.
I separate acknowledgement from confirmation. “Let me check that” is safe before execution. “You’re booked” is allowed only after a successful result. If the tool is slow, the agent can use a bounded acknowledgement once, then wait. Repeating filler makes the experience feel less reliable, not more.
Interruption handling must share the same state. If the caller interrupts an acknowledgement, the operation may still be valid. If the interruption changes a required parameter before execution begins, the pending request should be cancelled. The speaking layer cannot decide that alone; it needs the coordinator’s state and the model turn identity.
Test the protocol, not the personality
An AI-to-AI phone call can look automated while remaining nondeterministic. Two models may take a different path on every run, hiding regressions behind plausible conversation. For repeatable coverage, I use scripted utterances or fixed audio fixtures for the caller side and reserve live model-to-model calls for exploratory testing.
The deterministic suite should cover more than the happy path:
- a finalized zero-argument tool request;
- an unfinished request with an empty buffer;
- a tool that exceeds its execution budget;
- a caller interruption before and after execution starts;
- a reconnect after decision finalization;
- a dependency failure with an unknown outcome;
- a replay that must not duplicate the side effect.
Audio artifacts matter. Logs can prove state transitions, but they cannot prove that the pacing, acknowledgement, and final confirmation sound coherent. Saving the rendered call audio gives reviewers a stable artifact alongside the event trace.
Human review still matters
Some workflows deserve a human boundary even when the protocol is solid. High-impact changes, ambiguous identity matches, and irreversible operations should surface a review or confirmation step appropriate to the channel. The goal is not to make every call slow. It is to spend friction where uncertainty or consequence is genuinely high.
The best voice systems I have worked on are not the ones with the most elaborate personalities. They are the ones that know what has finalized, what has executed, what remains uncertain, and what they are allowed to say about each state.
What I would carry forward
Treat streamed tool use as a protocol with explicit finality, ownership, deadlines, and replay rules. Keep acknowledgement separate from confirmation. Test with deterministic caller inputs and save audible artifacts, not only transcripts. Most importantly, make uncertainty visible inside the system before trying to hide it with smoother language.