let's say someone writes in on a saturday. "your chat said you come out to westmere." you don't.
or the bot quoted a call-out price and left the weekend fee off. the customer is annoyed, and they have a point.
retune is free if you run your own copy, which runs on your own computer with an agent that builds what you ask for.
a bot answers from its prompt and the pages and files it read. change a few words in the prompt and answers you never touched can move.
that's the hard part. you edit the opening line so the bot sounds friendlier. a week later it's wrong about your service area, and nothing connects the two.
most people test a bot by typing a few questions into it the day it goes live. after that, nobody asks the old questions again.
the company in the video is oakridge plumbing. i made it up to show the feature.
the owner has ten questions on the tests tab, each with what a good answer has to include. she runs them. nine of ten. the burst pipe answer never mentions the $95 weekend fee.
then she shortens her prompt so it reads friendlier. the tests run as soon as she saves. six of ten, and three of those used to pass.
the broken ones sit at the top. asked about westmere, the bot now says yes, they come out. they don't cover westmere.
she puts the old wording back and adds the weekend fee. ten of ten. the chart keeps every run, so she can see exactly when it broke.
you write down the questions your bot must never get wrong. for a plumber that's the service area, the call-out fee, the weekend fee and what happens with a gas leak. you write what a right answer has to say, and anything it must never say.
then one click runs them all. a second ai model reads each answer and marks it pass or fail, with the reason. if auto-run is on, saving a new prompt runs them for you, and the prompt page shows the score.
for the customer in front of you today, fix the prompt or the page the bot read, and tell them plainly. then click "turn into a test" under that wrong answer in your conversations. that question gets checked from then on.
the tests tab isn't switched on for new accounts by itself. sign up, then email me and say you want tests on your bot. the steps, one by one, are in how to test a chatbot before launch.
it checks the questions you wrote. the bot can still be wrong about something nobody wrote down.
the grader is an ai model too. on a borderline answer it can pass once and fail the next time, so read the failures before you change anything.
auto-run starts when you save the prompt. if you retrain the bot on new pages, run the tests yourself.
the grading runs on openai, so you need an openai api key in your settings.
if the wrong answer came from an old page, like last year's prices, tests won't change what the bot read. when your chatbot gives outdated information is about that one.
when a retune bot doesn't know, it says so, and the question goes to an inbox where you write the answer once. when your chatbot keeps saying i don't know shows that inbox.
the tests cover the answers you can't afford to get wrong. they get checked every time you save the prompt.
it fits a business where a few facts carry the risk: an area you cover, a fee, an emergency rule. plumbers, electricians and repair shops are the obvious ones. an ai chatbot for a plumbing company is the version for them.
it helps less if your bot mostly answers open questions with no single right answer.
if your bot isn't retune, the idea still works: write the questions down and ask them again after every change. the tests tab is retune's way of doing that.
if it's useful, take it. if it's missing something, tell me and i'll build it.
olive