Agent testing

GET /v1/agents/{name}/test-runs#

List test runs. The agent's test runs, **newest first**, full transcripts included.

Parameters

Responses

Example request

curl "https://api.voiceland.ai/v1/agents/{name}/test-runs" \
  -H "Authorization: Bearer VL_API_KEY"

A successful response returns a AgentTestRunsResponse.

Example response

{
  "items": [
    {
      "channel": "voice",
      "id": "00001754550000000000000",
      "pass": false,
      "scenario_name": "book then cancel",
      "status": "failed"
    }
  ]
}

POST /v1/agents/{name}/test-runs#

Run a test scenario (synchronous). Executes one scenario against the agent's stored spec over the text path, the agent's own system prompt, model and tool surface, no live call, and returns the stored run. Pass scenario_id to run a stored scenario, OR an inline scenario (same shape as authoring) for an ad-hoc run that stores only the run. **Tools run for real where real is safe.** The calendar builtins (check_availability, book_meeting, find_bookings, manage_booking, reassign_meeting) execute against your actual calendar service, the same booking path a live call uses, so a passing scenario proves the booking flow, not a mock of it (find_bookings defaults to the scenario's caller_phone, as a live call defaults to the caller ID). knowledge_search is recorded but answers a static empty result; custom webhook/flow tools are recorded but never dial out. **The call is synchronous** and bounded (turn count, per-turn tool-hop cap and timeout): the response is the finished run, the full transcript with each turn's reply, every tool call with its arguments and result, the per-turn failure list and pass/fail verdict, and the aggregate status (passed | failed, or error when the harness itself broke mid-run; an errored run is still stored, but asserts nothing about the agent).

Parameters

Request body

application/json Schema: AgentTestRunRequest

Responses

Example request

curl -X POST "https://api.voiceland.ai/v1/agents/{name}/test-runs" \
  -H "Authorization: Bearer VL_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
  "scenario_id": "ats_9d8c7b6a"
}'

Example response

{
  "agent": "front-desk",
  "channel": "voice",
  "id": "00001754550000000000000",
  "pass": true,
  "scenario_id": "ats_9d8c7b6a",
  "scenario_name": "book then cancel",
  "status": "passed",
  "tenant_slug": "acme",
  "turns": [
    {
      "pass": true,
      "reply": "I can offer Monday 09:00 or 10:00, which suits you?",
      "tool_calls": [
        {
          "args": {
            "event_type": "et1",
            "member_id": "m1"
          },
          "name": "check_availability",
          "result": "Open start times (UTC, RFC3339): 2026-08-17T09:00:00Z, 2026-08-17T10:00:00Z."
        }
      ],
      "user": "I'd like an appointment tomorrow morning."
    }
  ]
}

GET /v1/agents/{name}/test-runs/{id}#

Get a test run. One run in full: per-turn user text (with {{last_uid}} already substituted), the agent's reply, every tool call with arguments and result, the failure list and pass/fail per turn, and the aggregate verdict.

Parameters

Responses

Example request

curl "https://api.voiceland.ai/v1/agents/{name}/test-runs/{id}" \
  -H "Authorization: Bearer VL_API_KEY"

Example response

{
  "id": "00001754550000000000000",
  "pass": false,
  "status": "failed",
  "turns": [
    {
      "failures": [
        "reply_missing: nine"
      ],
      "pass": false,
      "reply": "We are open until five.",
      "user": "When are you open?"
    }
  ]
}

GET /v1/agents/{name}/test-scenarios#

List an agent's test scenarios. Every stored test scenario for the agent, in scenario-id order. A scenario is a scripted multi-turn conversation, an ordered list of user turns, each with the expectations the agent's handling of that turn must meet, plus a channel (chat or voice) that shapes how it runs.

Parameters

Responses

Example request

curl "https://api.voiceland.ai/v1/agents/{name}/test-scenarios" \
  -H "Authorization: Bearer VL_API_KEY"

A successful response returns a AgentTestScenariosResponse.

Example response

{
  "items": [
    {
      "agent": "front-desk",
      "channel": "voice",
      "id": "ats_9d8c7b6a",
      "name": "book an appointment",
      "tenant_slug": "acme",
      "turns": [
        {
          "expect": [
            {
              "tool": "check_availability"
            }
          ],
          "user": "I'd like an appointment tomorrow morning."
        }
      ]
    }
  ]
}

POST /v1/agents/{name}/test-scenarios#

Author a test scenario. Stores one scenario: a name, a channel (chat or voice, voice runs add a phone-call preamble and advertise the full tool surface; chat runs drop the SIP-only builtins exactly as a widget session does), an optional caller_phone, and 1 to 20 turns. caller_phone sets the number the scripted caller is calling from: the run's preamble tells the model, and the find_bookings tool defaults to it, exactly as a live call defaults to the SIP caller ID. Set it to script "change my appointment" flows where the caller never recites a booking reference. Each turn is a user utterance plus optional expect assertions, all of which must hold: - tool, at least one tool call with this name fired on the turn. - args_contain, narrows tool: each value must appear (substring) in the matching call's argument of that key. - tool_result_contains, narrows tool further: the matching call's result text must contain this (case-insensitive). - reply_contains / reply_not_contains, case-insensitive substring assertions on the turn's final text reply. - no_tool, the turn fired no tool at all. The literal {{last_uid}} in a turn's user text is substituted at run time with the booking reference the run's last successful book_meeting returned (empty before any booking), how a script cancels or reschedules a booking whose reference the calendar mints.

Parameters

Request body

application/json Schema: AgentTestScenarioRequest

Responses

Example request

curl -X POST "https://api.voiceland.ai/v1/agents/{name}/test-scenarios" \
  -H "Authorization: Bearer VL_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
  "channel": "voice",
  "name": "book then cancel",
  "turns": [
    {
      "expect": [
        {
          "args_contain": {
            "member_id": "m1"
          },
          "tool": "check_availability",
          "tool_result_contains": "Open start times"
        }
      ],
      "user": "I'\''d like an appointment tomorrow morning."
    },
    {
      "expect": [
        {
          "args_contain": {
            "action": "cancel"
          },
          "tool": "manage_booking"
        }
      ],
      "user": "Actually cancel {{last_uid}}."
    }
  ]
}'

Example response

{
  "agent": "front-desk",
  "channel": "voice",
  "id": "ats_9d8c7b6a",
  "name": "book then cancel",
  "tenant_slug": "acme"
}

POST /v1/agents/{name}/test-scenarios/generate#

Generate test-scenario drafts with AI. Asks a model to draft test scenarios for this agent, grounded in the agent's own stored configuration, its system prompt, primary language (the drafted user turns are written in it) and tool surface. prompt is optional guidance ("cover cancelling and rescheduling"); count is how many drafts to ask for (1 to 8, default 3). **The drafts are returned UNSAVED.** Nothing is stored by this call: review the drafts, edit what needs editing, and save the keepers via POST /agents/{name}/test-scenarios, each draft's body is already the authoring shape (name, channel, optional caller_phone, turns with expect assertions), so a keeper posts as-is. Drafts the model produced that failed scenario validation are discarded and counted in dropped. Generation rides the same per-tenant LLM dispatch as a test run, on the agent's own model; the tokens are quality-assurance cost, metered on the grading bucket, not the conversational one.

Parameters

Request body

application/json Schema: AgentTestGenerateRequest

Responses

Example request

curl -X POST "https://api.voiceland.ai/v1/agents/{name}/test-scenarios/generate" \
  -H "Authorization: Bearer VL_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
  "count": 3,
  "prompt": "cover cancelling and rescheduling an appointment"
}'

A successful response returns a AgentTestGenerateResponse.

Example response

{
  "dropped": 1,
  "items": [
    {
      "caller_phone": "+306912345678",
      "channel": "voice",
      "name": "cancel via caller id",
      "turns": [
        {
          "expect": [
            {
              "tool": "find_bookings"
            }
          ],
          "user": "Θέλω να ακυρώσω το ραντεβού μου."
        },
        {
          "expect": [
            {
              "args_contain": {
                "action": "cancel"
              },
              "tool": "manage_booking"
            }
          ],
          "user": "Ναι, αυτό είναι, ακυρώστε το."
        }
      ]
    }
  ]
}

DELETE /v1/agents/{name}/test-scenarios/{id}#

Delete a test scenario. Removes one scenario. Idempotent, an absent id answers 204 too. Past runs keep their own copy of everything they executed, so run history is unaffected.

Parameters

Responses

Example request

curl -X DELETE "https://api.voiceland.ai/v1/agents/{name}/test-scenarios/{id}" \
  -H "Authorization: Bearer VL_API_KEY"

Schemas#

AgentTestGenerateRequest#

AgentTestGenerateResponse#

AgentTestRunRequest#

AgentTestRunsResponse#

AgentTestScenarioRequest#

AgentTestScenariosResponse#

Error#

Error envelope returned for non-2xx responses.

Expectation#

Run#

RunCost#

Scenario#

TestRunBillingResponse#

ToolCallRecord#

Turn#

TurnResult#