Every team that turns Kestrel on asks the same question in the first week: when can we stop checking its work?
Our answer is always the same number. Watch the first hundred runs in a category, then decide. Here is why that number, and what to look for while you watch.
Start with one category
Pick the category your team finds most repetitive. For most teams that is refunds for duplicate charges, or password resets. Do not pick the most common category if it is also the most varied. You want a hundred tickets that look alike.
Turn Kestrel on for that category only, with a low limit, and set it to ask before it acts.
What to check in each run
Open the run and read three things:
- What it read. Did it find the earlier messages from the same customer? An agent that misses context gives confident wrong answers.
- What it looked up. Did the facts it found match what the customer said? Look for the cases where they did not, and check that it stopped.
- What it wanted to do. Would you have done the same thing?
You do not need to read the reply closely at first. If the three things above are right, the reply almost always is.
The pattern in the first hundred
Across the teams we have watched, the first hundred runs tend to go the same way. The first twenty turn up one or two rules nobody had written down, such as "we never refund after ninety days". You add the rule. The next thirty are quiet. Somewhere around run sixty, someone on the team stops opening every run and starts opening only the ones that asked a question.
By run one hundred, the team has usually approved every proposed action for two weeks straight. That is the point to let it act alone inside the limit.
Then raise the limit, slowly
Do not jump from two hundred dollars to two thousand. Raise it once, watch another fifty runs, and raise it again. The limit is there to match how much you trust what you have seen, and that only grows by seeing more.