We’ve spent the past two years helping companies connect their documentation to AI assistants and Model Context Protocol (MCP) servers. We also built Biel.ai, so we see the same question as consultants and as builders: how do we make the answers better?
Teams often start by changing the model or rewriting the prompt. That can help, but many weak answers begin earlier. The assistant retrieved the wrong passage, pulled an outdated version, lost the page context during ingestion, or had no reliable source to work with.
These are the checks we make first.
1. Write sections that survive retrieval
A reader sees the page around a paragraph. A documentation assistant may receive only that paragraph and its heading.
Make each section understandable when it is retrieved on its own. Name the product, feature, or task in the heading. Restate prerequisites that change the meaning of the instructions. Replace headings such as “Configure it” or “Next steps” with headings that say what is being configured and what happens next.
You do not need to repeat the whole page. You need enough context for one section to remain accurate after it has been separated from the page.
2. Start with real user questions
Support tickets, site search terms, sales calls, and assistant logs show how people describe their problems. Use that language to check whether the docs contain a direct answer.
The page does not need to use a question as its title. It does need to cover the question users actually ask. “Authentication” may be a tidy section name, but users are more likely to ask why their token expired or how to rotate a key without downtime.
3. Show the complete procedure
Code examples need imports, configuration, file paths, and any setup required to run them. Procedures need prerequisites, expected results, and the important failure cases.
This matters because the assistant may retrieve the code block without the paragraph three sections earlier that explains where it belongs. A complete example gives the model fewer gaps to fill and gives the reader something they can verify.
4. Use one term for each concept
Choose a preferred term and use it consistently. If “API key,” “access token,” and “credential” mean different things in your product, the docs should make that distinction clear.
Keep known synonyms in a glossary, redirects, search configuration, or retrieval metadata when your system uses them. Do not scatter synonyms through the prose only to improve matching. That makes the documentation harder to read and can blur distinctions the assistant needs to preserve.
5. Curate what goes into the index
Do not add every page you own by default. Marketing pages, old release notes, duplicate migrations, archived versions, and internal drafts can compete with the current documentation.
Decide which sources are authoritative for each product and version. Exclude duplicate or obsolete content unless users still need it, and label historical material clearly when you keep it. This is indexing or retrieval scope, not model training.
6. Make versions and applicability obvious
A correct answer for the wrong version is still a wrong answer.
State which product, version, deployment model, plan, operating system, or SDK a section applies to. Put that information close to the instructions instead of relying only on a selector elsewhere on the page. When the same procedure covers several variants, say what changes for each one.
Archive or redirect unsupported versions when possible. If older documentation must remain available, keep it clearly separated from the current set.
7. Preserve context when pages are split into chunks
Most retrieval systems divide a page into smaller chunks before indexing it. A section called “Limits” becomes ambiguous if the chunk no longer carries the page title, heading path, product, or version.
Inspect the content after ingestion as well as the source page. Each chunk should retain the context needed to identify it. That may include the page title, parent headings, product, version, and a short description of what the section covers.
Anthropic describes a related technique in its guide to contextual retrieval: adding chunk-specific context before content is indexed.
8. Use metadata for decisions the system can enforce
Metadata is most useful when the retrieval layer can filter on it. Product, version, deployment model, audience, lifecycle status, and access level are practical fields because they can stop the wrong source from reaching the model.
Do not add tags and assume they improve retrieval automatically. Confirm that the ingestion pipeline preserves them and that search applies them. For example, OpenAI’s retrieval tools support attribute filtering before semantic search.
9. Keep important information in text
Some models can interpret screenshots and diagrams, but documentation pipelines do not all ingest them in the same way. A critical instruction that exists only inside an image is easy to lose.
Explain the required action, value, or decision in the surrounding text. Use alt text to describe the purpose of the image, and add a caption when the reader needs context. For a diagram, summarize the relationship or sequence the diagram is meant to show.
10. Give the ingestion pipeline a clean source
Find out what the assistant actually reads. It may crawl the published site, ingest Markdown from a repository, use an API, or receive selected resources through an MCP server.
If it crawls the site, make the main content available as clean HTML and use stable, crawlable URLs. Google can render JavaScript, but it still recommends server-side or pre-rendering because not every bot runs JavaScript. If the assistant ingests the repository instead, test the generated text after parsing and remove navigation, cookie banners, and other page furniture.
11. Scope by product, version, and access
One assistant can cover several products when the system can identify the user’s context and filter the sources reliably. Separate assistants or MCP servers make more sense when the products have different owners, audiences, or access boundaries.
Do not split the system per product as a reflex. First decide which content a user should be able to reach and which questions genuinely need information from more than one product. For restricted MCP resources, apply authorization and least-privilege access. The official MCP guidance recommends scopes tied to specific tools or capabilities.
12. Require sources and a safe fallback
An answer should point back to the documentation that supports it. Source links let readers check the details and help your team diagnose why an answer went wrong.
Decide what the assistant should do when the evidence is missing, conflicting, or outdated. A clear “I couldn’t find this in the documentation” is more useful than a plausible guess. It should also tell the reader where to go next, such as support, a product owner, or the relevant documentation area.
13. Build an evaluation set from real questions
Collect questions from support, search, onboarding, and assistant logs. Include common tasks, edge cases, ambiguous wording, version-specific questions, and questions the assistant should refuse to answer.
For each question, record the expected source and the criteria for an acceptable answer. Keep the first set small enough to review by hand, then add new cases as users expose gaps. OpenAI’s evaluation guidance for Q&A over documentation recommends combining production data, expert-written answers, and historical logs.
14. Test retrieval and the final answer separately
When an answer fails, check the retrieved sources before changing the prompt.
If the correct section never appeared, investigate the content, chunking, metadata, filters, or search configuration. If the correct source appeared but the answer was still wrong, look at the instructions, model, and answer-generation step. Measuring both stages prevents a prompt change from hiding a retrieval problem.
Run the evaluation set against a staging index whenever you change the corpus, chunking, retrieval settings, prompt, or model. Keep the previous results so you can see what improved and what regressed.
15. Start with a bounded pilot and classify every failure
You do not need perfect documentation before testing an assistant. You do need a clear scope. Start with one product, audience, or workflow where the sources are reviewed and users have another route to get help.
Log weak and unanswered responses, then classify the cause:
The answer is not documented
The source is outdated or contradictory
The correct source was not retrieved
The wrong product or version was selected
The user did not have the right access
The model misread good evidence
The first two clearly belong in the documentation roadmap. The others point to retrieval, scoping, access, or answer generation. Some failures will need changes in more than one place.
If you are unsure where to begin, run ten questions that your support team answers every week. Check the sources the assistant retrieved alongside the prose it generated. You will know what to fix next.
We built Biel.ai around this approach. Its content-gap analytics show what users are asking, which questions keep coming up, and where your documentation needs work.