Test your agent before it goes live

An agent that answered well last month can start answering badly after a prompt edit or a knowledge change. The evaluation page catches that before your visitors do: you keep a set of test questions, run them against the agent draft, and decide what a failing run does to a deploy.

Build the golden set#

Open Evaluation in the console and pick an agent. Each case is a question plus at least one expectation:

• the expected answer, judged by meaning rather than exact wording,
• an answer the agent must never repeat (a confirmed past mistake),
• or how it must behave: cite your documents, hand over to a human, or decline to answer.

A case with no expectation cannot fail, so the editor will not save it.

The fastest way to a useful set: press Import confirmed-wrong reports. Every answer your reviewers confirmed wrong on the Answer feedback page becomes a case the agent must never repeat, with the visitor's original question attached. Importing twice is safe: existing cases are skipped, never overwritten, so your edits survive.

Run it#

Run the set asks every question to the agent draft, with its own instructions and knowledge, and an automatic judge grades every answer. The stored run shows a pass or fail for each case, why it failed, and what changed since the previous run: which cases started failing, which recovered, which were added.

Runs are kept forever, so "what did this version pass before it went live" always has an answer.

Decide what a failure means#

The deploy gate is a project-wide setting with three positions:

• Off: runs are informational.
• Warn: a deploy over a failing run succeeds, but tells you which run failed.
• Block: a deploy is refused until the run passes or the gate is lowered.

Only a genuinely failed run gates. A run that errored because the harness itself had a problem never blocks your deploy.

Watch quality in production#

The same page holds the project's quality alarms: a ceiling for the hallucination rate (answers not backed by your documents, measured by an automatic judge on a sample of real conversations) and for the human takeover rate. When a rate crosses its ceiling, an alert event is sent to your configured channels, and another when it recovers. The dashboard shows the same numbers: grounded answers, negative feedback, and chats resolved without a person.

The periodic operations report#

Operations report in the console is the same quality report for a whole calendar period, kept as a record rather than as a live dial.

Turn on the schedule, pick every month or every quarter, and add the addresses that should receive it. At the end of each period the report is generated and emailed as a spreadsheet. It is also kept here, so you can read it or download it without waiting for the email. With no addresses it is still generated and kept, just not mailed to anybody.

Each report opens with the headline it is built around: the share of website chats that ended without anyone asking for a person, with the target next to it, met or not. Under it comes the whole project, then the same set of numbers per agent: conversations handled, how many were resolved without escalation, how many answers were backed by your documents, negative ratings, how the conversations felt from start to end, the busiest topic, how many caller requests matched an intent you configured, and the evaluation runs of the period.

You do not have to wait for a period to end. Generate produces a report for any past month or quarter on demand, and offers two buttons: generate it without sending, to read it first, or generate and send it to the addresses on the schedule.

Generating on demand is limited to a handful of reports per hour. One of them reads every conversation, rating and evaluation run in the period before it answers, and emails a spreadsheet unless you asked it not to, so the limit is what keeps a few clicks from becoming a load on the same records your live conversations are using. Working through a quarter's three months in one sitting is well inside it. If you do reach it the console says so and asks you to wait a few minutes; your scheduled report is never affected, because the limit is on on-demand generation only.

Every number in the report is the same computation the dashboard and your own monitoring read. A period that hit a read limit says so on its own row: the numbers are then a lower bound over the most recent conversations rather than the whole period.

When the score moves and nobody touched the agent#

Turn on Drift detection beside the deploy gate and your golden set is re-run for you: on a schedule you set, and again whenever the models behind your calls change. Each run is compared with the previous one for the same agent, and you are alerted when the score falls further than the limit you set.

It alerts on a drop, not on a floor, and that is the difference that makes it usable. An agent that has always scored 60 percent on a deliberately hard set is not drifting. One that scored 92 yesterday and 84 today is, and only the second is worth waking somebody for.

The alert goes to the same channels as your quality alarms, and names the cases that passed last time and fail now. A re-run says on the runs list why it happened: on schedule, or because the models changed.

It is off by default because each re-run costs one agent answer and one grading pass per question in your set.