Evaluating an AI leasing agent

When a rental inquiry arrives, the agent decides to reply, stay quiet, or hand the thread to a person. This page shows how that decision is measured: each stage, the data it uses, and the name the stage usually goes by.

The project is in progress. The counts are real (353 test cases from 143 email threads), and every example is made up because the real ones hold applicant details.

The map

Hover or tap any stage or dataset. A stage lights up the data it uses; a dataset lights up the stages that use it.

1Cold start

aka bootstrapping, before there is anything to measure

There are no outputs and no labels yet. What do we evaluate on?

2Offline evaluation

aka pre-deployment evaluation, development evals

Is this version good enough to ship?

One square is one test case.annotated so farnever run

Repeat until new annotations reveal no new failure modes

3Online evaluation

aka post-deployment evaluation, production monitoring

Is it still good on real traffic, and what is it getting wrong now?

Back to offline evaluation, with new test cases

We are hereOffline evaluation

Improve

aka iterate, prompt engineering

Take the largest failure mode and change one thing. Run the experiment again on dev, and check that its count fell and no other rose.

The thing changed is most often the prompt: the system prompt or skill, and any few-shot examples in it. But anything that shapes the output can be the fix: the context the prompt is built from, tool definitions and their descriptions, the output schema, the model and its settings, or the code around the call.

Where the fix goes depends on the failure. A missing fact is a data problem. A rule with an exact answer belongs in code. Only judgement belongs in the prompt, sometimes as a few-shot example from the train split.

Uses: train split, dev split.

In this project

The first fix from error analysis made things worse: a rule to escalate when our own earlier message conflicts with the case fixed two fails and broke six decisions that had passed. The labels on passing decisions caught it, and two narrower versions followed.

Which layer owns the failure

“Didn't know the unit has parking” → data “Offered a time inside the notice period” → code “Too stiff”, “should have waited” → prompt “Asked for a discount” → none. Escalating is correct.

The whiteboard version

The same process, condensed enough to redraw from memory.

Cold start

inputs → baseline prompt → dataset → split into train | dev | test

Offline

on dev: experiment → annotate → error analysis → improve ↺

then: evaluators → judge alignment → one run on test

Online

release behind approval

tracing → feedback + online evaluators → trace review → dataset growth ↺

new test cases flow back to offline

What the first experiments showed

Every experiment ran on the same 30 test cases, with one kind of change between each. In the first, most escalations were the agent asking for facts it had not been given, so the fix was to the data. The third followed a line-by-line check of the prompt against the instructions it replaced, and the fourth a decision to offer tours through a booking link. The counts alone do not say whether the replies are good. Annotation of the fifth does: 23 of 26 decisions passed. Later experiments were scored by code against those 26 labels, which is how the sixth was caught making things worse.

Experiment	What changed	Reply	Stay quiet	Escalate	Annotated
Experiment 1	Baseline prompt. Listings had rent and lease dates only.	12	6	12	No
Experiment 2	Added a fact sheet per unit and standard answers.	20	5	5	No
Experiment 3	Policy checked against the instructions it replaced: decline a low self-reported credit score, restored tour rules.	19	5	6	No
Experiment 4	Tours offered through a booking link by default; times only for an already-booked tour or a time the lead named.	20	4	6	No
Experiment 5	Answers that are the same for every lead moved from the case into the prompt.	19	4	7	26 of 30: 23 pass
Experiment 6	First fixes from error analysis. The new escalation rule fired on almost any earlier message.	11	2	17	Scored against labels: 19 of 26
Experiment 8	Escalation and confirmed-tour rules spelled out case by case.	17	4	9	24 of 26
Experiment 9	Same prompt on a newer, cheaper model.	15	4	11	23 of 26
Experiment 10	Booking line reworded to say "in-person showing".	18	3	9	24 of 26
Experiment 11	No change. First run on a second sample of 30 dev cases.	13	5	12	No
Experiment 12	No policy change. Tour state set by hand in 6 cases of the second sample, and only those run again.	17	6	7	30 of 30: 26 pass
Experiment 13	Fixes from the second error analysis: a clarifying question leads the reply, a turned-down time is not offered again, an accepted time gets one confirming reply.	20	3	7	No
Experiment 14	A lead who names a time we do not have gets the booking link, not a list of the closest times.	20	3	7	No
Experiment 15	A time less than 24 hours away is not available, even when we offered it. First run graded by the LLM judge.	21	2	7	By the judge: 20 of 30 pass
Experiment 16	Five policy questions answered by the owner directly, with a default offered for each, and three exact rules moved into code checks.	21	3	6	By the judge: 25 of 30 pass
Experiment 17	Confirming run before merge. A detail the fact sheet marks as not on file is escalated, not answered with a no.	21	3	6	By the judge: 28 of 30 pass

About the names

The field has not settled on one vocabulary. Each stage above carries the name used most often, with the others listed as aka. The same check is an evaluator, a grader or a scorer depending on the tool, and the same split is dev or validation depending on the author's background.

Train, dev and test come from machine learning and are the usual names wherever splits are discussed. The published advice applies them to the annotated data used to build an LLM judge. Using the same three splits while tuning the agent's own prompt is this project's extension of that reasoning.

Two different things therefore get split: test cases, to tune the agent, and later the annotations made on dev, to tune the judge. Both splits use the names train, dev and test.

Cold start is the least standard name here. The sources describe the phase, usually as what to do before there is production traffic, without agreeing on what to call it.

Sources

AI Evals: Everything You Need to Know, Hamel Husain and Shreya Shankar
Error analysis, binary annotations, synthetic data, train / dev / test for a judge, offline and online evals, guardrails.
Using LLM-as-a-Judge, Hamel Husain
One domain expert, a written critique with every verdict, true positive and true negative rates.
Product Evals in Three Simple Steps, Eugene Yan
Label, align the evaluator on dev, check it on test; one evaluator per dimension.
Demystifying evals for AI agents, Anthropic
Tasks, trials, graders and transcripts; capability evals that graduate into a regression suite.
Evaluation concepts, LangSmith
Dataset, example, experiment, evaluator, reference output; how offline and online evaluation feed each other.
Evaluation best practices, OpenAI
Where eval data comes from, calibrating automatic scoring against people, evaluating on every change.