Tests

A test is a conversation the platform holds with your agent and then grades. Run your tests against a draft before you publish it, and you find out whether a prompt or tool change broke something that used to work.

Tests run over chat. Chat builds the same agent a phone call does (prompt, model, tools, and knowledge), without speech recognition and text-to-speech. A test with a Caller Number runs with the call wording of the platform skills. A test without one runs as a website chat, where collect-code and verify-identity use chat wording and read-back-values is skipped. Tools that only exist on a call run against stand-ins the test sets up: a caller number and contact, keypad entries, workflow variables, and a caller-memory record (see Call inputs). A test never reads or writes your real contacts, what the agent remembers about a real caller, or a dropped call, and never switches to another agent.

Settings that act on the audio of a call aren't covered: When the caller goes quiet, Thinking cue, Allow barge-in, Voice activity detection, Turn detection, and Maximum call duration. Call recording tools can't be tested either.

Open your tests

Open an agent and select Tests in the editor header. The Review step shows the status of the version on screen as a badge:

• Passing 12/12 — 12 of the 12 tests passed on this version as it is now. When the first number is lower, the rest haven't run on this version yet or couldn't run on it.
• 3 failing — three tests failed or ended in an error on their latest run.
• Running 4/12 — a run is going, and 4 of its 12 attempts have finished.
• Not run — the tests have never run on this version.
• Stale — they ran, but the version changed since.

An agent with no tests shows no badge.

The Tests page opens on the version you came from. Pick another in the Version list to see its results; the published version is marked (published). Tests belong to the agent and aren't versioned, so every version runs the same tests.

The Agents list has a Tests column with each published agent's result on its published version, using the same badges as above; select a result to open that agent's Tests page. A published agent with no tests shows No tests, which opens its Tests page too, and an agent that isn't published shows a dash. The column is hidden first in a narrow window.

Write a test

Select Add Test. Give the test a Name and, optionally, a Description, then pick a Caller:

• Scripted — sends the Caller Messages you write, one at a time, each after the agent replies. Use it when you know exactly what a caller says.
• Simulated — has an AI play the caller. You give it a Persona (who the caller is and what they know) and a Goal (what they want). It uses only the facts in the persona, and it talks to your agent until it decides the goal is reached or can't be reached, or until Max Turns runs out. Use it to try the conversations you didn't think to script.

To change a test, select Edit on its row. Duplicate and Delete are in the row's menu. At 50 tests, Add Test and Duplicate are unavailable.

Tool responses

Tools don't run for real in a test unless you ask. Each tool answers with an empty result, or with the JSON you enter as its Response under Tool Responses. A test can't send a text message or book an appointment by accident that way, and a test of your booking flow doesn't depend on what's on the calendar today.

Turn on Call for real for a tool to run the actual tool. The editor warns you that the tool can change data outside the test unless the platform knows it's read-only: an integration tool that sends a GET, a Zendesk tool set to Show Ticket, or get_current_time. A test with a tool on Call for real isn't run again after an interruption, because the tool may already have run; it ends in an error instead.

capture_dtmf, set_workflow_variable, and transfer_to_voice_agent answer from the test's call inputs unless you give them a response, and they aren't marked Stubbed. update_contact needs a Contact, and lookup_contact a Caller Number. With Call for real they run against the test's contact, never a real one, and lookup_contact finds only that contact. A tool marked Call for real that reads the caller's number, such as an integration tool that uses {{sys.telephony.ani}}, receives the test's Caller Number, so it can reach the real system for that number.

Some tools can't take a response because they run inside the agent. end_call, search_knowledge_base, skills, and agent tools always run for real. The linked agent behind an agent tool runs too, and its own tools answer from your tool responses like any other tool. A workflow handoff can't take a response either. It ends the test without running the workflow, and a successful transfer_to_external or transfer_to_voice_agent ends the test the same way, as Handed off. The other agent doesn't run, so give it tests of its own. A transfer the agent is refused, such as to an agent with no published version, doesn't end the test.

The Tool picker lists the version's tools. To name a tool the list doesn't show, type its name. That is how you reach a tool of a consulted agent, named after the agent tool, two underscores, and the tool (billing__lookup_invoice), or a tool from an MCP server.

Name a tool the way the version's tool list shows it. Saving refuses a name that isn't a tool, such as a setting's name (transferToVoiceAgent for Transfer to another agent) or, when you save against a version, a spelling of one of its tools with different capitals or underscores (FindSlots for find_slots), and says which tool you meant. Saving also checks tool and disposition names against the version you have open, so a test written for another version may be refused with "is not configured on this version".

Call inputs

Under Caller Messages, Add Keypad Entry adds keys the caller presses: 0 to 9, *, and #, up to 32. Keypad entries right after a message are what the caller types when the agent's reply calls capture_dtmf, in order. A capture_dtmf with no entry left gets nothing, as when a caller presses no keys. An entry no capture took is sent as its own turn, the way a call delivers keypresses with Keypad Input on: only its first group, ending at the agent's end key (which isn't sent) or its maximum digits. Without Keypad Input it's ignored. A keypad entry that follows another keypad entry is a turn of its own, so it can't answer a capture_dtmf that the earlier one triggered. A Response you set for capture_dtmf stubs the tool, so it takes no entries. Every queued entry is then sent as a keypad turn after the message (or ignored, without Keypad Input). In a test the capture_dtmf prompt is only a tool argument and isn't spoken, so a Reply contains check on that prompt doesn't see it, and a criterion sees it only as the tool call's argument. set_workflow_variable is offered in a test whenever it is enabled, but on a call only when the agent resumes a workflow at the end.

These sections of the editor apply to scripted and simulated tests alike:

• Caller Number — the caller's Phone Number, in E.164 (+15551234567). It fills {{sys.telephony.ani}}, and the contact and memory belong to it. Leave it blank for a caller with no number.
• Contact — the Contact switch makes the agent find a contact for the caller number, with the Attributes you add (each a Name and a Value). It needs a Caller Number, and contacts turned on for the agent. The contact exists only in the test and nothing is saved to it. On a call to a contacts-on agent, every caller with a number has a contact. To test a first-time caller, turn Contact on and add no attributes.
• Caller Memory — the Caller Memory switch makes the caller a returning one, with Prior Calls and up to three notes: Synopsis (read at the Warm memory depth), Summary of Prior Calls (Light and Continuity), and Open Items (Continuity). It needs a Caller Number and a memory depth other than None. Only a custom returning-caller greeting applies, as on a call. A rewritten greeting doesn't, because the test reads the base greeting. With the switch off, memory is off during the test.
• Workflow Variables — values the calling workflow passed in, read as {{vars.name}}, each a Name and a Value. set_workflow_variable can set only a variable the version declares, and the Sets a workflow variable check picks from those.

Nothing real is touched. The contact and memory record are built for each turn from the test; no real contact is read or changed, no caller-memory record is read or written, and no dropped call is filed. Saving a test is refused when an input can't be used:

• Tool "update_contact" needs a caller contact. Turn on Contact.
• Tool "lookup_contact" needs a caller number. Set one under Caller Number.
• Contacts are off on this version, so the test's contact is never read.
• Caller memory is off on this version, so the test's memory record is never read.
• Keypad entries need Keypad Input or keypad capture on this version.
• Workflow variable "<name>" is not declared on this version.

Checks and criteria

A test passes only when every check and every criterion passes.

Checks are rules the platform evaluates in code. Add one with Add Check and pick its Type:

A regular expression with a quantified group inside another quantifier, such as (a+)+, is refused when you save. A pattern that still takes too long to evaluate fails its check.

Criteria are plain-language expectations, such as "Confirms the new appointment time before ending". Add one with Add Criterion. An AI judge grades each one against the transcript and gives a reason. A criterion the transcript can't answer fails with the reason "no evidence".

Prefer a check when one can express what you expect. A check gives the same answer every time for the same conversation.

Repeat

An agent doesn't answer the same way every time. Set Repeat to run a test up to 5 times. It passes only when every attempt passes, and its row says how many attempts passed, such as "2 of 3 attempts passed". Leave Repeat blank to use Default Repeat from the test settings.

Recommend tests

Select Recommend Tests, beside Add Test and in the empty state, to ask for up to 3 suggested tests for the version you have selected. They're written from that version's setup (prompt, tools, transfers, and dispositions) and up to 20 of the agent's most recent calls. An agent with no calls yet gets tests from its setup alone.

The button reads Writing tests… while it works, usually under a minute. When it finishes, a toast says how many tests it added, "No new tests to suggest", or "Couldn't write tests. Try again." If 10 suggestions are waiting for the agent by the time it runs (coaching, for example, can add one while you wait), it says "This agent already has 10 suggestions waiting. Accept or dismiss some first." instead. The tests land under Suggested Tests below, on the same page.

Recommend Tests is unavailable while 10 suggestions are already waiting for the agent; accept or dismiss some first.

Make a test from a conversation

Open a finished call or chat in Insights → Call Logs and select Make a Test. The button shows only for a conversation a voice agent handled. It opens that agent's Tests page with a new scripted test in the editor, named after the conversation:

• Caller Messages — what the caller said, in order.
• Tool Responses — each tool the agent called, answering with the last result it returned that wasn't an error. A tool that only returned errors, or a tool that can't take a response, gets none.
• Checks — a Calls a tool check for each tool the agent called, and an Ends with a disposition check for the disposition the conversation ended with.

A draft can include capture_dtmf, set_workflow_variable, and transfer_to_voice_agent calls. It leaves out update_contact and lookup_contact, which need a Caller Number and Contact you add yourself.

The test isn't saved until you select Save, so trim it first: drop the caller messages and tool responses the test doesn't need, and add the criteria a good answer meets.

A draft holds up to 40 caller messages and 20 checks, and stays small enough to save. When the conversation has more, the draft keeps the earlier messages and tells you so. A website chat keeps only its last 60 messages, so a draft from a long chat starts partway through it.

Suggested tests

Suggestions appear under Suggested Tests, at the top of the agent's Tests page, and on the agent's page in Automations → Agents → Coaching. One list, two sources: Recommend Tests above, and coaching, which can also suggest tests built from the calls it reviews, up to 2 per review. Unless the agent has automatic reviews on, a coaching review reads only the calls a supervisor coached or took over, so that's where its suggestions come from. After you apply a coaching recommendation, it suggests one more test that checks the change holds; that card is labeled Guards an applied change. On the Tests page, a suggestion from coaching is labeled From Coaching, so you can tell the two sources apart.

Each card gives the reason for the test and, when one is cited, a quote from the call it came from. Accept opens the test in the editor: in place on the Tests page, or on the agent's Tests page when you accept from Coaching. Saving it creates the test and marks the suggestion accepted; leaving without saving keeps it waiting. Accept All, on the Tests page, saves every waiting suggestion as a test. A suggestion the server can't save (for example, because it would pass 50 tests) stays waiting with its error shown on the card, and the rest are saved. A toast reports how many saved, such as "Saved 2 of 3". Suggested tests never turn on Call for real, and Accept All leaves waiting any older suggestion that does, so you review it in the editor first. Dismiss clears a suggestion, and the toast that follows carries an Undo.

No suggestions are added while 10 are waiting for an agent. Suggestions come from voice calls only, because coaching reviews calls only, and they're kept for 90 days.

Write tests with Tandy

In the Tandy chat, type /writing-agent-tests and describe the test you want, or ask for a test from the call or chat you have open. Tandy reads the agent and its existing tests, writes the test, and shows it on a card naming the agent, the test, the caller, and how many checks and criteria it has. Create saves it.

Tandy can also run the agent's tests and read you the results. A run uses model tokens, so Tandy asks for your approval first. It works through the tools listed under the Tandem MCP server, so an assistant you connect yourself can do the same.

In the test list, a test Tandy wrote shows a Tandy badge, and one saved from a suggestion shows Suggested.

Run tests

Select Run All to run every test on the version, or the play button on one test's row. One run per agent goes at a time. While it runs, the page shows how many attempts have finished, such as "1 of 5 attempts finished", and Cancel Run stops it. Run All and each row's play button do nothing until it ends, and say "A run is in progress" when you point at or tab to them. When a run finishes while you're on the page, a message summarizes it in one of three ways: "All 5 tests passed" (or "1 test passed"); "2 of 5 tests failed", which counts Failed and Error tests (an error counts as failed here, as it does in the editor badge); or, when no test failed but some didn't pass, a warning such as "4 of 5 tests passed". A test that is Invalid, didn't finish, or was edited or deleted during the run doesn't count as passed. View Results in the message lists every test in the run, and the × closes it. A cancelled run shows no message, and neither does a run that finished before you opened the page.

Each test's row shows its result from the newest run on this version that included it, so running one test keeps the others' results. A run that ended before it reached a test, because it was cancelled or couldn't start, leaves that test's earlier result in place. Editing a test clears its result, and the row reads Not run until the test runs again. A result is one of:

• Passed or Failed — every check and criterion passed, or at least one didn't.
• Error — the test couldn't finish, for a reason that isn't your agent failing it: a model didn't respond, the test ran past its time limit, or the version was edited during the run. The result says which. An error counts as failing in the editor badge.
• Invalid — the test names a tool or disposition this version doesn't have, uses a tool without the call input it needs, or sets a call input the version doesn't read (a contact with contacts off, memory with memory off, keypad entries with neither Keypad Input nor keypad capture, a Sets a workflow variable check on a variable the version doesn't declare). It didn't run. This includes a Tool not called check on a tool the version doesn't have, which can never fail.

Each row's badge shows the result. Under the test's name, a test that repeats says how many attempts passed, such as "1 of 3 attempts passed" (or, while it runs, how many finished), then how long ago the newest attempt finished, such as "5m ago". To open the result, select the row, or select View Results in the row's ⋮ menu. The result shows each check and criterion with its outcome and reason, then the transcript. The row you opened is highlighted. A criterion is marked AI judged, because a model graded it; a check is evaluated in code. Tool calls that didn't run for real are marked Stubbed. A tool that ran for real shows the same status chip and request and response tabs as any other transcript row. Show Turn under a criterion highlights the turn the judge based it on. A test that repeated shows one Attempt button per attempt, each marked passed or failed, and opens on the first attempt that didn't pass. While a test is running, its row opens the run in progress, which says how many attempts are still running, and results appear as attempts finish. A test that hasn't run on the version opens with Run Test. The check and criterion wording comes from the test as it is now, so in Run History a test you edited after the run shows its new wording beside the old outcomes. In a narrow window the results cover the test list; Esc or the × closes them.

Run History lists the version's earlier runs, each with how many attempts passed, such as "3 of 5 attempts passed". Select one to see every test in it, pick a test for its results, and select All Tests to go back to the list.

Before you publish

The publish dialog shows the version's test results. When every test passed, it says so. When tests are failing, stale, or running, when they haven't run, or when some of them didn't run on this version, it warns you and labels the confirm button Publish Anyway. It never stops you from publishing.

The warning offers two buttons. Run Tests starts a run on the version and opens the Tests page. View Tests opens the Tests page without starting a run. While this version's tests are running, only View Tests appears. An agent runs one test run at a time, so if a run on another version is going, Run Tests says a run is already in progress and opens the Tests page without starting a new run.

Test settings

Select the gear beside the Version list:

• Simulated Caller Model — plays the caller in simulated tests.
• Judge Model — grades criteria against the transcript.
• Agent Temperature — overrides the agent's temperature during tests only, from 0 to 1. A low value reduces variance between runs; it doesn't make them identical. OpenAI models, and Bedrock models with reasoning on, ignore it. Leave it blank to use the agent's own.
• Default Repeat — how many times a test runs when it doesn't set its own Repeat.

Limits and defaults

Through the API

Every action on this page is also available through the API. See List an agent's tests and the other Agent tests operations in the API reference.