Synthetic data · replayed runs
An AI account advisor for customer success.
Retention Agent
Reviews account usage and support signals, recommends a next step, and drafts outreach for human approval.
- 40recorded runs
- 4models
- 6tools
- scored on right play, facts traced, follows playbook
Recorded runs
Which customers might not renew, and what should we do about each one?
That’s the question an account manager faces every week. I built an AI agent that does the prep: it checks the customer’s numbers, reads the company playbook, recommends one action, and drafts the email. A person decides. I ran it on ten example customers with four different AI models, and recorded every step so you can watch.
Explore recorded agent runs on synthetic customer data and inspect the supporting evidence.
How one run works
- Check the customerUsage, unused seats, support tickets
- Read the playbookThe rules for each situation
- Recommend one actionOut of five possible plays
- Draft the emailUsing only real numbers
- A person decidesApprove or hold. Nothing is sent.
What it does for one customer
- 19secfrom customer name to a drafted email (median run)
- <1¢per customer with the best model
- 10/10right action, best model
- 94%of numbers in the drafted emails matched the data
Measured across the 40 recorded runs. A person still reviews every draft.
Pick a model and a customer
All ten customers are invented. The numbers are synthetic.
- medium risk
Show the data
Fields
segment SMB plan Team seats_purchased 40 arr 8,191.64 renewal_date 2027-01-04 days_to_renewal 111 risk_score 46.2 risk_band medium top_drivers usage_decline, idle_seats - −32% usage
Show the data
Summary
weeks_of_history 12 prior_weeks 8 recent_weeks 4 prior_avg_active_seats 37 recent_avg_active_seats 25.2 usage_drop_pct 31.8 prior_avg_sessions 127.8 recent_avg_sessions 89 sessions_drop_pct 30.3 Week by week
week_ending period active_seats sessions feature_events 2026-06-30 prior 38 124 1,138 2026-07-07 prior 36 117 987 2026-07-14 prior 37 125 1,130 2026-07-21 prior 36 137 1,298 2026-07-28 prior 38 126 1,199 2026-08-04 prior 37 130 1,062 2026-08-11 prior 38 128 1,199 2026-08-18 prior 36 135 1,165 2026-08-25 recent 25 88 698 2026-09-01 recent 26 96 864 2026-09-08 recent 25 91 843 2026-09-15 recent 25 81 671 - 38% idle
Show the data
Fields
seats_purchased 40 recent_weeks 4 recent_avg_active_seats 25.2 idle_seats 15 idle_seat_pct 37.5 arr 8,191.64 arr_per_seat 204.79 idle_seat_arr 3,071.86 - support healthy
Show the data
Fields
open_tickets 4 escalations 0 csat 4.2 recent_ticket_topics notification settings, mobile sync delays, billing question - 2 lookups
Show the data
Lookup 1: “adoption workshop 30% active seats drop guidance outreach”
Adoption workshop Play type: adoption_workshop. Use it when the account's previously active seats have gone quiet: a usage drop of 30% or more, measured as the average active seats over the last 4 weeks against the average of the prior 8 weeks. The people who bought the product are still there; they have simply stopped opening it, often after a team change, a busy quarter, or a workflow that moved elsewhere. A workshop for those existing users is the answer, not a commercial change. Seat rightsizing Play type: seat_rightsizing. Use it when usage is steady but the account is paying for far more seats than it uses: at least 40% of purchased seats have been idle across the last 4 weeks, and active seats have not dropped sharply (a drop under 30% versus the prior 8 weeks). The account is not unhappy; it is over-provisioned, and an honest conversation now beats a surprise cancellation at renewal. Adoption workshop Do not mention seat counts, pricing, or the renewal date in the first message. Do not name an individual; write to the whole team. Keep the draft to two or three short paragraphs and one concrete ask: a date for the session. Lookup 2: “adoption workshop message structure session length what to offer”
Adoption workshop Suggested subject: "A short working session for the {account} team". Adoption workshop Do not mention seat counts, pricing, or the renewal date in the first message. Do not name an individual; write to the whole team. Keep the draft to two or three short paragraphs and one concrete ask: a date for the session. Adoption workshop The offer is a short, hands-on working session for the existing users, not a sales call. Propose a 45-minute workshop built around the two or three features the account used most when usage was healthy, and ask which workflow fell away. Quote the actual weekly numbers from the usage tool so the team can see the same picture you see. - matches the expected play
Show the data
Cited numbers
claim field value active seats dropped from 37 to 25 prior_avg_active_seats 37 active seats dropped from 37 to 25 recent_avg_active_seats 25.2 weekly sessions fell from around 128 to 89 prior_avg_sessions 127.8 weekly sessions fell from around 128 to 89 recent_avg_sessions 89
The proposal
Adoption workshop
right actionWhy: Usage down 32%, 38% of seats idle and support healthy.
To: the Dunmoor Legal Logistics team
From: Your account manager
Subject: A short working session for the Dunmoor Legal Logistics team
We've noticed that active seats dropped from 37 to 25 over the past month, and weekly sessions fell from around 128 to 89. Your team was getting great value earlier this summer, and we'd like to help you get back to that momentum.
We'd like to offer a 45-minute hands-on workshop focused on the features your team used most when engagement was strong. This isn't a sales call—it's a working session to understand which workflow may have shifted and how we can support your team's day-to-day needs.
Can we schedule a session in the next two weeks?
Highlighted numbers were checked against the data the agent looked up. 4 of 4 check out.
Your choice stays on this page. Nothing is sent.
Same ten customers, four AI models
Which model would I use? GPT-6 Luna.
It picked the right action for all ten customers, in about 18 seconds and a tenth of a cent each. The most expensive model wasn’t the most accurate.
- GPT-6 Luna (the model I would use)10/10 right actions0.1¢ · 18 s per customer
- Claude Sonnet 4.59/10 right actions9¢ · 33 s per customer
- GPT-4o mini8/10 right actions0.1¢ · 10 s per customer
- Claude Haiku 4.58/10 right actions3¢ · 19 s per customer
Each square is one customer: filled means the model picked the action the company’s own rules call for. Cost and time are per customer.
Problem
Customer success teams need to identify accounts that may not renew and decide which signals deserve a conversation.
Challenge
Let a model decide which account signals to check and which play to propose, then show whether its draft sticks to the data it retrieved, without treating risk indicators as proven churn predictions.
My role
I built the recorder, the six tools, the five playbooks, the evals and this replay.
Architecture
- 1Synthetic dataset: 60 accounts with weekly usage, seats, support tickets and renewal dates
- 2Python recorder calling each model through OpenRouter with OpenAI-compatible tool calling
- 3Six tools the model chooses between: account_risk, usage_drop, idle_seats, support_summary, retrieve_docs (BM25 over five playbooks) and propose_play with cited facts
- 4Four models run on the same ten accounts; every run is validated against a JSON schema
- 5Evals on each run: play match, facts traceable, follows playbook
- 6This page replays the recorded runs and ends at an approve-or-hold step; nothing is sent
Technologies
Approach
- Give the model narrow tools and let it choose the order; the recorder keeps every call, its arguments, the result and the latency.
- Ask for a source on every number, percentage and date in the draft, then check each one against the recorded tool result.
- Record every model against the same accounts and seed, and replay the runs here without calling a model.
- Stop at a proposal: approving or holding it in the replay changes nothing outside this browser tab.
Current result
40 recorded runs across four models and ten accounts, each replayed above call by call with its draft and evals. Play match against the dataset’s expected play: Claude Haiku 4.5, 8 of 10; Claude Sonnet 4.5, 9 of 10; GPT-4o mini, 8 of 10; GPT-6 Luna, 10 of 10.
Lesson
Recording real runs shows what a scripted demo hides: runs that stalled because my recorder cut playbook passages short and the models kept searching for the rest, and drafts whose numbers sound right but do not match the data. Checking each cited figure against the tool result is what makes a draft quick to review.
