maybe a temp or a bot once said "yes, we cover that" when you don't. then a van goes out for nothing, or a customer waits for a call back you can't give.
retune is free if you run your own copy, which runs on your own computer with an agent that builds what you ask for.
so this letter is less about the bot and more about how you check it.
the company in the video is oakridge plumbing. i made it up to show the feature.
if your site gets asked the same few things, most of them have one right answer, and a wrong one costs you. these are the ten i set up for oakridge:
yours will differ, but the shape is the same. where you go, what it costs, what counts as an emergency, and what the bot must not walk someone through.
each bot gets a tests tab. you add a question and write what a good answer has to include, one fact per line. you can also write what it must never say, like "yes, we cover westmere".
for the gas stove you tick should refuse. the right answer there is "call a licensed gas fitter", not steps.
you click run tests. the bot answers each question the way it would answer a customer, from its prompt and the pages it read. a second ai model reads each answer and marks it pass or fail, with the reason.
in the video the first run is nine of ten. the burst pipe answer never mentions the $95 weekend fee.
then the owner shortens her prompt so it sounds friendlier. the tests run by themselves when she saves. six of ten, and the bot now says yes to westmere.
she puts the old wording back, adds the fee, and gets ten of ten.
the rule i'd keep: change nothing on the bot you haven't run the tests on. auto-run does that for prompt edits. the step by step is in how to test a chatbot before launch.
a visitor opens the chat on your website and types a question. the bot answers from your prompt and the pages it read. if it's 11pm and a pipe has burst, it gives them what you wrote for emergencies, like your after-hours line and the fee.
it can ask for a name and an email or phone number, so you can call back. if most of your calls come in after you've closed, when customers call and you're already closed is about that.
there's no plumbing template. you write your hours, your service area and your fees into the bot's prompt, and point it at your website so it reads your pages. the ten questions above make a good checklist for what the prompt needs.
the tests tab isn't switched on for new accounts by itself. sign up, then email me and say you want tests on your bot. once it's on, the tab shows up next to your others.
the grader is an ai model too. on a borderline answer it can pass once and fail the next run, so read the failures yourself.
it only checks the questions you wrote. the bot can still be wrong about a town or a job nobody put in a test.
auto-run starts when you save the prompt. if you retrain the bot on new pages, click run tests yourself.
the grading runs on openai, so you need an openai api key in your settings.
when a retune bot doesn't know, it says so, and the question goes to an inbox where you write the answer once. the tests cover the other half: the answers you can't afford to get wrong.
if it already happened to you, my chatbot gave a customer the wrong answer is about that.
if it's useful, take it. if it's missing something, tell me and i'll build it.
olive