a docs bot that quietly skipped a section answers as if the section isn't there. so here's how retune reads your docs, and how you see what it missed.
it's free if you run your own copy of retune. that means retune runs on your own computer, with an agent that builds what you ask for. email me for that, or sign up and start on retune today.
the site in the video is maple street bike shop. i made it up to show the feature. it's a small site, but a docs site goes the same way, just with more rows.
you give it your docs domain or your sitemap url. it finds the sitemap, follows it into any smaller sitemaps, and lists every page as a row you can tick.
you narrow it down with patterns. /docs/** in include keeps only the docs, and the ** means everything under that path. /blog/**, /changelog/** in exclude drops the rest.
if you keep old versions, like /v1/ and /v2/, include the one you want the bot to answer from. one crawl takes up to 500 pages.
then you hit crawl and watch. every page is its own row, going from crawling to done, with its title once it's read. the count at the top says how many are done and how many failed.
failed pages go to the top of the list, each with the reason. a server hiccup says "try again", and that's what retry failed is for. a page that's really gone says "page not found" and stays flagged. so you end up with a list of broken links in your own sitemap.
when you're done, hit ask the bot and ask it something from a page deep in the docs.
it reads the html your server sends. it doesn't run javascript. docs that ship as plain html pages come through fine. docs that load their text in the browser after the page opens come out thin.
here's a quick way to tell. open one of your docs pages, view the page source (not the inspector) and search for a sentence from the page. if it's there, retune can read it.
it only reads public pages on your own domain. docs behind a login don't work. pdfs and images in the sitemap are skipped, so an api reference that's only a pdf won't come through.
it doesn't follow the "disallow" lines in robots.txt. if there's a section you don't want in the bot, like internal notes that happen to be public, exclude it.
past 500 pages, crawl again. pages it already read show as already trained, so the next crawl takes the rest.
what it stores is the text of the pages it read, in your bot in retune. it doesn't change anything on your docs site.
new pages are easy. run discover again and only the new ones are ticked.
edits to pages it already read don't come through that way. for that there's a sync tab. it re-reads pages on a schedule and shows a diff of what changed.
the sync tab follows the links from a page you pick, not the sitemap, so you'd add each section's index page. the steps are in keep your chatbot updated with website changes.
if nothing in your docs covers a question, the bot should say it doesn't know instead of guessing. it can still get one wrong, so ask it the questions you get most before it goes live.
the questions it couldn't answer can land in one inbox, grouped. for someone who keeps docs, that's a list of pages you haven't written yet.
and if your users work in claude code, the bot can also answer them there as an mcp server. that's tested with claude code. claude desktop and chatgpt weren't tested.
reading a whole sitemap isn't switched on for new accounts by itself. sign up, then email me at olive@reframe.so and say you want the sitemap crawl on your bot. once it's on, a "from a sitemap" section shows up in the train tab.
olive