Testing Voice and WhatsApp Agents Before Go-Live

Most writing about AI evaluation is aimed at product teams with data scientists. Small businesses launching a phone receptionist or a WhatsApp booking assistant need something simpler: a checklist they can run in a few days that catches the failures customers would otherwise find first.

This is the test plan I use before a voice or messaging agent takes real traffic. It works whichever platform the agent runs on; I compare the main voice platforms in Vapi vs Retell vs ElevenLabs.

Why do AI agents need different testing from normal software?

AI agents need different testing because the same input can produce different outputs. A normal web form either works or it does not; an agent may handle a booking correctly nine times and misunderstand it the tenth. Testing has to measure how often it works, not whether it worked once.

Retell's documentation on simulation testing says it directly: "Judge a scenario on its pass rate across runs, not one run", because both the simulated caller and the grader are themselves language models. Hamel Husain's widely read guide to LLM evals makes the same point from the other side: "you don't necessarily need a 100% pass rate" on every check, but you do need to keep looking at real conversations, because "no free lunch exists."

So the plan has two halves: repeated, automated scenario runs to measure pass rates, and hands-on testing of the things simulation cannot catch.

Step 1: write the scenario set

Start with 20 to 50 scenarios written from real customer conversations: call recordings, inbox history, or the questions your staff answer most. Each scenario needs a description of the customer and a pass criterion that a person could check.

Write one behaviour per criterion. Retell's guide advises: "Write one behavior per criterion, and be specific about what counts as success." Vague criteria such as "was helpful" produce test results nobody trusts.

Step 2: run each scenario many times in simulation

Simulation means an AI plays the customer, following your scenario description, while your agent responds as it would on a live call. Run every scenario several times, because a single pass tells you little.

Three practices make simulation useful:

• Mock the tools. Retell describes function mocks that intercept a call "and returns a set result instead of calling the real function", so tests do not create real bookings and return the same result every time. Mock failures too, to test the tool-failure scenarios.
• Agree the bar per scenario. A wrong answer to an opening-hours question costs little; a wrong booking costs a customer. Set a stricter pass rate for scenarios that change data or money.
• Rerun after every change. Any prompt edit, model switch or new tool can break a scenario that used to pass. The scenario set is your regression test, the same idea as giving a coding agent a way to check its work.

If your platform lacks built-in simulation, a small script that sends scripted messages and checks responses works for WhatsApp, and Hamel's "Level 1" idea applies: simple assertions, "like you would write in pytest", that run fast and cheaply. My post on why AI agent projects fail covers evals in more depth.

Step 3: test what simulation cannot catch

Text simulation does not test the phone line or the messaging channel itself. Make real calls and send real messages before launch, from different phones and networks.

For voice agents, check:

• Response delay. Measure the gap between the caller finishing and the agent starting to speak. Human conversation runs on very short gaps: a study of turn-taking across ten languages found "a minimization of silence between conversational turns", with average gaps varying only "within a range of 250 ms" of the cross-language mean. Long pauses feel broken to callers even when the answer is right.
• Interruptions. Talk over the agent mid-sentence. It should stop and listen.
• Noise and audio. Call from a car, a street, a speakerphone.
• Numbers and names. Dates, phone numbers, spellings and addresses read back correctly.
• Voicemail and silence. What happens when nobody speaks, or a voicemail answers an outbound call.

For WhatsApp agents, check:

• Message templates. Meta's template documentation says template messages "are the only type of message that can be sent to WhatsApp users outside of a customer service window", and templates must be approved before use; review "can take up to 24 hours". Test every reminder or follow-up your agent sends later, and submit templates early.
• Media and formats. Voice notes, photos, locations and long messages. Decide what the agent does with each.
• Bursts. Customers often send three short messages in a row. The agent should not answer each one separately.

Step 4: drill the handoffs and failure paths

Handoffs and failures are where small-business agents lose customers, so test them deliberately. Every agent needs a reliable way to reach a person, and a sensible fallback when a tool breaks.

• Ask for a human in the first sentence, in the middle and at the end. The agent should transfer or offer a callback each time.
• Check that the person receiving the handoff sees a summary, so the customer does not repeat everything.
• Switch off a tool (calendar, CRM) during a test and confirm the agent takes a message rather than inventing a booking.
• Confirm high-impact actions need confirmation. OWASP's guidance on excessive agency recommends human approval for "high-impact actions before they are taken"; for a small business that usually means refunds, cancellations and anything involving payment.

Step 5: soft launch, then keep reading transcripts

Go live on a small share of traffic first: one phone line, certain hours, or a fraction of WhatsApp conversations, with a person ready to step in. Widen it once the transcripts look right.

After launch, read a sample of transcripts every week. Hamel's guide recommends having "humans periodically evaluate at least a sample of traces", and that is the cheapest quality control a small business has. Every failure you find becomes a new scenario in the test set, so it cannot quietly come back.

Keep an eye on cost while you test as well: simulation and test calls use real minutes. I worked through the per-minute numbers in AI voice agent cost per minute.

Work with me

I build and test voice and WhatsApp agents for small businesses, including the scenario set, simulation runs and a staged launch. See voice AI agents and AI chatbot development, or browse my work and book a scoping call.