AI Voice Agents for Customer Service: A Build Guide

I compared the main platforms in Vapi vs Retell vs ElevenLabs. This post covers the part that decides whether a voice agent succeeds regardless of platform: what to automate, how to design the call, what the rules require, and how to test it.

Why voice agents are moving into customer service now

Analysts expect service to be the first large-scale home for agentic AI. Gartner predicts that by 2029, agentic AI will autonomously resolve 80% of common customer service issues without human intervention. Note the word "common": the prediction is about routine issues, not every call.

Builders moved early. Andreessen Horowitz's 2025 voice agent update reported that companies building with voice made up 22% of the most recent Y Combinator class, citing Cartesia, and noted that in December 2024 OpenAI cut its realtime API price by 60% for input and 87.5% for output. The same piece makes a point I agree with: enterprises rarely switch from fully human to fully AI call handling at once. Voice agents win a small share of calls first and expand from there.

Step 1: Pick the call types, not "the phone line"

The most common scoping mistake is "automate our phone line." A phone line carries dozens of call types with different difficulty and risk. Pull a sample of recent calls or call reasons and sort them.

Good first candidates:

• Booking, rescheduling and cancelling appointments.
• Order or case status ("where is my order", "has my claim been received").
• Business information such as hours, locations, documents required.
• Lead qualification with a short, fixed set of questions.
• After-hours intake that captures details and creates a ticket for the morning.

Leave for later: complaints, billing disputes, anything involving vulnerable callers, and calls where the right answer depends on judgement.

Step 2: Design the conversation around actions

A voice agent is an AI agent with a phone as its interface. Its value comes from what it can do in your systems during the call: check a calendar, look up an order, create a ticket, send an SMS confirmation. Each action is a tool with a narrow, typed interface, and the same rules apply as for any agent:

• Separate read actions from write actions. Looking up an order is low risk. Issuing a refund is not.
• Confirm before writing. Read back the date, time and name before booking. Speech recognition errors are normal; confirmation catches them.
• Verify identity before revealing personal data. Decide what the caller must confirm before the agent reads out anything private.
• Log every action with the call ID, so any call can be reviewed.

Step 3: Keep it fast and interruptible

Callers notice pauses far more than chat users do. In practice:

• Keep the agent's turns short. One question at a time.
• Allow interruptions. Callers talk over agents, and the agent should stop and listen.
• Say something useful while a slow action runs ("Let me check that booking for you") rather than going silent.
• Run tool calls in parallel where possible, and keep the context the model sees small.

Measure latency on real phone lines, not on a laptop microphone. Network conditions and telephony add delay you will not see in the browser.

Step 4: Make the human handoff a feature

The handoff is where most voice agents frustrate people. Design it deliberately:

• Clear triggers: the caller asks for a person, the agent is not confident, the topic is on a "always escalate" list, or the same question has failed twice.
• Warm transfer with context: the human should receive a summary of the call so far, so the caller does not repeat themselves.
• A fallback when nobody is available: take a message, create a ticket with the details and give the caller a realistic callback time.

Step 5: Follow the disclosure and consent rules

Regulation already applies to voice agents, and it differs by market. This is an engineering summary, not legal advice.

EU: tell callers they are talking to an AI

Under Article 50 of the EU AI Act, systems that interact directly with people must be designed so those people know they are dealing with an AI, unless that is obvious, and the information must be given clearly at the latest at the first interaction. These transparency duties have applied since 2 August 2026 and were not delayed by the 2026 amendment. For a voice agent, the disclosure belongs in the opening line.

US: AI voices count as "artificial" voices for outbound calls

In February 2024 the FCC issued a declaratory ruling that AI-generated voices are "artificial" voices under the Telephone Consumer Protection Act, which means calls using them are subject to the same consent requirements as other artificial or prerecorded voice calls. If your voice agent places outbound calls, consent management is part of the build, not an afterthought.

Everywhere: recording and data

Check call-recording consent rules for each region you operate in, decide how long you keep recordings and transcripts, and restrict who can access them. I work with clients across the US, UK, Canada, Germany, the UAE, Australia and India, and the answer differs between them. A short legal review before launch is cheaper than a redesign after.

Step 6: Test like it is a product

Before any real caller hears the agent:

• Script test calls for every call type, including awkward ones: background noise, accents, callers who change their mind, callers who give information out of order.
• Test the handoff paths, including when no human is available.
• Build an eval set from transcripts. For each test call, check that the right actions were taken, the confirmation happened and the disclosure was given. The method is the same as for any agent; see why AI agent projects fail, and the evals that prevent it.
• Launch on a slice of traffic, such as after-hours calls or one call type, and review a sample of real transcripts every week.

What I have built

My AI voice agent for education, built with Next.js and the Gemini API, lets students speak instead of typing: speech is transcribed, structured into organised notes and made queryable. It is not a call-centre agent, but it taught me the lesson every voice project shares: the transcript is an intermediate step, and the value is in what the system does with it. The same holds for customer service. A voice agent that transcribes perfectly but cannot book the appointment has not helped anyone.

Work with me

I build voice agents that connect to real systems, disclose properly and hand off cleanly. See voice AI agents and AI agent development, or book a scoping call and bring a list of your most common call types.