Most docs sites are not set up for this. Some block AI crawlers by default in robots.txt and don’t know it. Others rely on JavaScript rendering that crawlers can’t see. Others have great content that buries the answer under three paragraphs of context.

Here’s what we’ve been testing and fixing across consulting projects.

Four tests to run now

Test 1: is your site crawlable without JavaScript?

Open your docs in a fresh browser. Disable JavaScript. Reload.

If the page loads and the content is readable, AI crawlers can probably read it too. If not, you have a problem. Most crawlers don’t execute JavaScript. They see what curl sees.

You can also test directly:

curl -A "ClaudeBot" https://your-docs.com/getting-started

If the response is a mostly empty HTML shell with a <script> tag, AI crawlers are getting the same empty shell.

Test 2: is your robots.txt blocking AI crawlers?

Check https://your-docs.com/robots.txt for these user agents:

GPTBot        # OpenAI
ClaudeBot     # Anthropic
PerplexityBot # Perplexity
CCBot         # Common Crawl (used by many LLMs)
Google-Extended # Google's AI training bot

If any of these are disallowed, AI crawlers are skipping your site. Some CDNs and hosting platforms block AI bots by default as part of “bot protection” settings. Worth checking.

Test 3: do you rank for questions users actually ask?

Pick three questions a developer might ask about your product. Google them with your company name.

Example: "biel.ai create account", "biel.ai docusaurus install", "biel.ai customize widget"

If you don’t rank in the first results, AI probably can’t find you either. AI retrieval uses the same signals as search engines, weighted differently. This applies both to live retrieval (what ChatGPT and Claude Code query in real time) and to training data (what the models learned from). No Google ranking usually means no AI ranking in either case.

Test 4: does your docs answer common questions up front?

Pick three common questions users ask about your product. Open the relevant docs page for each.

Is the answer on the page? Is it in the first paragraph, or buried three paragraphs deep?

AI agents retrieve a chunk of content and return an answer based on it. If your page opens with context before the answer, the agent will summarize the context instead. If the answer is missing entirely, the agent will either hallucinate or say it doesn’t know.

The 90%: indexable + useful + popular

Those four tests measure three things:

• Indexable: can AI crawlers access your content?

• Useful: does your content answer what users ask?

• Popular: is your content referenced by high-signal sources?

Cover these three and you’ve done 90% of the work.

The 10%: extra credit

We can’t know exactly which retrieval methods each AI system uses. The more signals we provide, the more discoverable our content becomes. If you’ve covered the basics and want to push further, here’s what to add.

Publish an llms.txt file. Like sitemap.xml but for LLMs. A plain text file at your site root describing what your documentation contains and how it’s organized.

# Biel.ai Docs
> AI chatbot for technical documentation

## Getting Started
- [Install in Docusaurus](https://docs.biel.ai/docusaurus)
- [Install in MkDocs](https://docs.biel.ai/mkdocs)

## API Reference
- [Authentication](https://docs.biel.ai/api/auth)

Still emerging, still evolving, but increasingly supported.

Serve markdown URLs. Crawlers and agents work best with clean markdown over bloated HTML. You can serve the same page at .md for a cleaner, token-efficient version. Example: yoursite.com/api.md instead of making them parse your HTML, CSS, navigation, and cookie banner.

Structured data and metadata. Add Schema.org markup for pricing, FAQs, and product info. Keep Open Graph tags and meta descriptions accurate. Use page metadata to describe what each page is about, including keywords and synonyms for concepts users might call by different names (e.g. “asset” vs “token”).

Publish machine-readable schemas. Make your OpenAPI specs publicly available. Same for GraphQL schemas, Protocol Buffers, or any structured contract your API exposes. Agents can parse these directly, no HTML scraping needed.

Measure where you stand

Fern’s Agent Score evaluates your docs against 22 checks covering most of the above. You get a grade and a list of issues, and a prompt you can paste into Claude Code to fix them automatically.

We ran it on our own docs and got a C on the first run. As they say, “the shoemaker’s son always goes barefoot”.

But this didn’t stop Biel.ai from ranking first when asking ChatGPT for the best AI chatbot for Docusaurus. Because we focused on the basics first.

Start with the basics

The 90% is where most of the value is. Be crawlable, improve the quality of your docs, make them popular. The extra 10% is worth doing if you’re already in good shape, but don’t skip the basics to chase it.

Most docs teams are still catching up. Start with the basics now and your docs will be easier to find when everyone else is still figuring it out.

We started following it. Within a few minutes, we had our answer. Some steps worked fine. Others didn’t compile. One snippet referenced a method that no longer existed. Another used a dependency version from two releases ago.

The client didn’t know. The page looked great. Clean formatting, syntax highlighting, clear explanations between each code block. Nothing about it screamed “broken.” You had to actually run the code to find out.

This is what happens when code lives in Markdown files, disconnected from any build process. Nobody re-runs examples after every release. The docs look fine. They just don’t work anymore.

Keep examples in a project, not in your docs

The fix: stop pasting code into Markdown files. Instead, keep your examples in a separate project that actually compiles.

The project lives in its own folder near the docs, with real dependencies, a real build step, and version control. Your docs import snippets from this project instead of containing them directly. When the product releases a new version, you update the project. If a snippet breaks, the build fails. You find out before your users do.

You can also version your examples properly. Need docs for v2 and v3? The project has branches for each. You bump the dependency, run the build, and know immediately what still works.

Once we set this up for the client, updating code examples stopped being scary. It went from editing a Markdown file and hoping it still worked to making a code change with a build. New users following the getting started guide actually got to the end.

It does mean maintaining a project alongside your docs. But the alternative is finding out your examples are broken when a user does.

If you want the practical details on how to set this up, I wrote a step-by-step guide a few years ago.

Across the industry, when companies need to reduce headcount, documentation teams are among the first on the list. Sometimes to cut costs, sometimes to justify AI investment. But the stated logic tends to be the same: AI can write docs now, so why pay people to do it?

Not every company is moving in this direction. And the ones that aren’t have a very different theory about what documentation is for.

The companies cutting writers

There are AI agents that detect code changes and draft doc updates. Tools that generate release notes, changelog entries, and API references from code and product specs. They’re fast, and for companies shipping quickly, that speed is a real advantage. On paper, you no longer need a dedicated team for this.

The question isn’t whether AI can produce docs. It’s whether it can maintain them, structure them, and make them useful over time without someone thinking about that full-time.

In practice, documentation gets reassigned to engineers or product managers, who use AI to draft it. In theory, it works. In reality, developers have already absorbed testing, infrastructure as code, CI/CD, on-call. Now add docs strategist to the pile. They have more on their plate, not less. Docs get written when there’s time, which is rarely.

Nobody reviews whether the content actually helps a user get from A to B. Nobody maintains the getting started guide when the SDK changes. Nobody decides what shouldn’t be documented.

And to be fair, there’s a real argument that the people who build the thing should document the thing. I don’t disagree. We’ve been advocating for that even before AI entered the picture. But there’s a difference between engineers contributing to docs and engineers being fully responsible for docs strategy, maintenance, and information architecture on top of everything else they already own. I’m describing a pattern I’ve seen, not citing a controlled study, but from the contracts we’ve picked up after layoffs like these, it’s consistent: the docs drift, support tickets go up, onboarding takes longer, and there’s nobody whose job it is to fix it.

Could this work with strong processes, dedicated doc sprints, or better guardrails around AI-generated content? Maybe. I haven’t seen it succeed at scale yet, but I’d welcome examples that prove otherwise.

The companies investing in writers

On the other side, some companies are hiring more writers, not fewer, and using AI to augment them. Their reasoning: documentation is now the source material that every AI system depends on. AI coding assistants, chatbots, support agents. They all consume your docs. If the docs are wrong, the AI is wrong.

Here’s a concrete example. An AI coding assistant trained on outdated API docs will confidently suggest deprecated endpoints and wrong parameter names. Developers follow the suggestion, hit errors, file issues, and lose trust in the tool. The root cause isn’t the AI. It’s the docs the AI was reading.

These companies treat documentation as infrastructure. The foundation for how users and AI understand their product. The better the docs, the better every experience built on top of them.

Anyone who’s actually used AI to write or code knows it’s a back and forth. You prompt, you review, you correct, you re-prompt. The output needs judgment. Someone who knows what good docs look like, not just someone who can generate more of them.

But it’s not just about producing docs. There’s also a role AI can’t fill, at least for now. Technical writing has never been 100% writing. It’s deciding what gets documented and what doesn’t. Knowing who to ask for the right information. Evaluating whether docs are useful or need rework. Figuring out what order things should be learned in. And knowing when adding more pages actually makes things harder to find. An AI will happily generate pages for everything. A good docs team knows when to say no.

Where I stand

I run a documentation engineering consultancy. I have skin in the game, and I’ll be honest about that. We’ve had contracts cancelled because companies decided AI could handle the docs. We’ve also gained contracts from companies that understand docs are the fuel for AI.

Our position: AI makes writers more capable, not redundant. Writers who use AI to draft, then edit, verify, test, and structure produce better output than before. Maybe the companies that cut their teams had other reasons for making that call. But creating better docs wasn’t one of them.

The companies investing in documentation right now understand something simpler: every answer AI generates is only as good as the docs it’s reading.

The assistant kept responding “I couldn’t find information about this” for basic things. Export data. Reset password. Delete account.

Support had been answering these verbally for years. Everyone assumed it was documented, but it wasn’t.

Why basic flows go undocumented

When a team ships something new, they want to document the shiny parts. The feature they spent months building. The complex integration that’s hard to understand without a guide.

Nobody writes a page about how to reset a password. It feels too obvious.

Then support starts getting questions. They answer them in 30 seconds over chat, and writing a whole doc page for that feels like overkill. So it stays verbal. Months pass, new support agents join, and they learn the answers from colleagues, not from docs. At some point everyone just assumes it’s written down somewhere.

An AI assistant breaks this cycle. It can only answer what’s documented. It doesn’t know what support told a user last week, and it doesn’t have access to Slack threads or old tickets. In our case, that’s intentional. We only train it on curated, reviewed documentation.

So when a user asks “How do I export my data?” and the assistant says “I couldn’t find information about this,” that gap is no longer invisible. It shows up in the logs, every day. You can’t ignore it.

Use the gaps

Once we documented the missing flows, tickets dropped. The assistant started answering questions it couldn’t before.

But here’s the part we didn’t expect: the “I don’t know” responses told us exactly what to write next. We just looked at what users asked most and started there.

AI assistants don’t fix documentation problems. They make them visible.

The client said it casually, like it was just how things worked. The system rebuilt the entire site every time. Even for a one-word fix.

This is more common than you’d think. And it kills documentation quality in ways that aren’t obvious.

Slow builds make docs worse

When a build takes 30+ minutes, nobody verifies small changes. You fix a typo, push it, and hope for the best. You don’t wait half an hour to check if the formatting looks right.

Writers stop iterating. They batch changes into big updates instead of making frequent small improvements. Reviews slow down because nobody wants to trigger another long cycle just to see a preview.

The result: fewer updates, more errors that slip through, and a team that treats docs as something to avoid touching.

Why this build was slow

The project used Sphinx. The build produced both HTML and PDF outputs. Every build generated everything from scratch: the full HTML site and the complete PDF manual.

In this case, the PDF was the bottleneck. Sphinx uses LaTeX to produce PDFs, and LaTeX is slow. One monolithic PDF meant one monolithic build. Change a paragraph in the authentication guide and the system rebuilds a 1400-page PDF from scratch.

On top of that, the HTML and PDF builds ran sequentially. One waited for the other. So even if the HTML build was fast, the PDF blocked the whole pipeline.

What we changed

Two things:

We split the PDF into chapters: Instead of generating one massive PDF, we configured Sphinx to produce separate PDFs per section. Authentication docs, API reference, deployment guide, each their own PDF.

Change the auth docs? Rebuild only that PDF. The rest stay untouched.

We separated the HTML and PDF builds: They now run in parallel instead of sequentially. Writers can preview their HTML changes immediately while the PDFs generate in the background.

The result

Build time dropped from 30 minutes to under 5 for most changes.

But the real impact isn’t the number. It’s what changed in the team’s behavior. Writers started verifying their changes again. Small fixes got pushed the same day instead of being batched for “next sprint.” Reviews got faster because previews were available in minutes.

The docs got better because the build time got out of the way.

Slow builds are a silent documentation killer. Nobody complains about them in sprint retros. But they quietly push teams away from touching the docs.

If your build takes more than 10 minutes, it’s worth investigating why.

Two years ago, they’d open your docs site and read the guide. Today, they type “How do I authenticate with this API?” in Claude Code. The AI answers, writes the code, and moves on. Your docs site never sees a pageview.

Top engineers at Anthropic and OpenAI report that AI now writes 100% of their code. If developers aren’t even writing code themselves, they’re definitely not browsing your docs to figure out how to write it.

We're seeing this in the documentation projects we work on. AI-driven traffic to docs sites is growing while human visits drop. Mintlify reports that almost half of documentation site traffic now comes from AI agents, not humans. Your docs have a new primary reader. It doesn’t browse. It queries.

From reading to asking

Documentation always competed with Stack Overflow, GitHub issues, and asking a coworker on Slack. AI coding assistants just won that competition.

When you can describe your problem in natural language and get working code back, in context, inside your editor, there's no reason to open a docs site.

But that doesn’t mean docs matter less. If anything, accuracy matters more now. A human reading a confusing paragraph will slow down, re-read, maybe search for other sources. An AI will just pick an interpretation and ship it.

The problem with public AI models

When a developer asks an AI about your product, the model either relies on training data or searches the web. Both are broken.

Training data is stale. If you shipped a new SDK version last month, the model doesn’t know. It’ll suggest deprecated methods with full confidence. Web search relies on SEO. The model might find a three-year-old Medium article instead of your current API reference. Your content’s discoverability depends on how well you rank, not on how good your docs are.

Neither approach has a special relationship with your documentation. It doesn’t know your custom error codes or the migration guide you published last Tuesday.

The result: AI generates code based on outdated examples and wrong parameter names. The developer doesn’t even know the answer was wrong until something breaks.

It gets worse for internal docs

If public docs have a discoverability problem, internal docs are completely invisible.

Your internal wiki, your Confluence spaces, your private Notion pages. AI models can’t see any of it. A new engineer joins your team, asks their AI assistant “How do I deploy to staging?”, and gets nothing useful. The model wasn’t trained on your deployment runbook. It can’t search your internal network.

Your internal documentation is invisible to every AI tool your developers use daily. That’s a real problem.

Connect your docs to the tools developers actually use

There’s a fix for this. Instead of hoping AI models happen to find your docs, you can connect your documentation directly to the AI tools developers are already using.

This is what the Model Context Protocol (MCP) enables. It’s an open standard that lets AI assistants connect to external data sources. Think of it as a bridge between an AI coding assistant and your documentation.

With an MCP server pointing at your docs, the assistant fetches answers directly from your content. Not from training data. Not from Google. From your docs, in real time.

And here’s the part most people miss: you register a description of what your docs contain. The AI uses that description to decide when to query your docs. If a developer is working with your SDK and hits an authentication problem, the assistant knows your docs are the right place to look. It’s not a keyword search. The AI decides based on context.

What this means for documentation teams

Your docs go from a website people browse to a data source AI tools query. That changes things.

Content quality matters more, not less. Every ambiguity, every outdated example, every missing step gets amplified. Bad docs produce bad AI answers at scale. Good docs produce answers that make developers trust your product.

Internal docs become reachable. That deployment runbook, your team’s coding standards, your architecture decision records. MCP can expose them to AI assistants inside your organization.

You also get better feedback. When AI assistants query your docs, you can see what questions are being asked. That’s direct signal about what developers are trying to do. Better than pageview analytics ever was.

What this doesn’t fix

MCP doesn’t fix bad docs. If your content is incomplete or outdated, AI will serve incomplete, outdated answers. Just faster.

And it doesn’t eliminate hallucinations. But it gets things wrong less when it has access to the right content. Your docs site is still the canonical source of truth. When AI gives a wrong answer, that’s where developers go to verify. MCP just makes your content reachable from more places.

Where to start

Start thinking about how AI assistants consume your content. Not how humans read it. How machines query it.

Structured content, clear headings, explicit examples. These matter even more now. Your new reader doesn’t browse pages. It parses them.

Look into how MCP can work for your docs. Connect it to the tools your team already uses. See what questions come in.

Your docs are already being consumed by AI. The question is whether you’re serving the right answers or letting the model guess.

It was. Four years ago.

Recently a client called about their docs site. Sphinx 4 versions behind, dependencies with security vulnerabilities, relying on deprecated extensions, theme broken on new browsers, search noticeably slower.

The launch got budget. Content creation got budget. Someone even set up CI to build and deploy automatically. The team celebrated. Then moved on.

Framework upgrades, dependency updates, security patches? Nobody even brought them up. There was no line item for “keep the docs site alive.” Because it was alive. It was working fine.

Your docs site didn’t break

You just stopped keeping up with everything around it.

Every dependency you don’t update becomes a risk. Every framework version you skip makes the next upgrade harder. None of it is urgent on any given day. All of it is urgent four years later.

The decay is invisible. The site loads. The search works. The pages render. Until one day they don’t. Or they do, just slower, just slightly broken, just enough for users to notice before you do.

The cost equation

Keeping a docs site maintained is routine work. Not always simple, but manageable if you stay current. It just never wins against everything else on your roadmap. A few hours a quarter to update dependencies. A CI check for broken links. An annual framework upgrade before you fall too far behind. Call it 20 to 30 hours a year. Maybe $3 to 5K if you’re paying someone.

That client's docs were broken for at least three months before anyone flagged it. The fix took another six weeks. Four and a half months of slow search and a theme that looked off on half the browsers visiting it. All that time, the docs were quietly telling users you don't maintain your tools.

Your docs site is a production system

We all understand this for apps. Nobody ships a production app and walks away for four years. There are monitoring tools, dependency bots, upgrade cycles.

Docs sites get none of that. They get launched and forgotten. Then someone calls you four years later wondering why everything is broken.

It has users, dependencies, and security surface area. It deserves the same care.

That client’s site is fixed now. They’re on a maintenance plan. The kind of work that never makes headlines but means nobody has to make that call again.

The review process was simple: build the docs locally, check for broken links, formatting issues, rendering problems. Then approve.

Except reviewers weren’t building the docs locally. They were skimming the markdown in the PR and hitting approve. “LGTM!” Without ever seeing the actual rendered output.

Can’t blame them. Building docs locally means cloning the repo, installing dependencies, running a build command, waiting, then navigating to localhost. That’s five minutes of friction for a review that “should” take two. So they skip it and pray it’ll render well.

This isn’t a discipline problem

It’s a friction problem.

We don’t skip important steps because we’re lazy. We skip them because the effort feels disproportionate to the task. A quick review shouldn’t require a local dev environment. But it did. So the “quick review” became “skim the markdown and approve.”

The fix was simple

We added automated PR previews. Every pull request now generates a deployed preview and drops the link in the PR comments.

“Click this link” replaced “build it locally.” One click versus five minutes of setup. Guess which one people actually do.

Review quality went up immediately. Not because we trained anyone, not because we wrote stricter guidelines, not because we added more process. We just removed the friction.

The pattern

This isn’t unique to doc reviews.

Style guides nobody follows because checking them is a chore → put linting in CI and suddenly everyone complies.

Broken link checks that never happen because clicking every link is tedious → automate it and they get caught before merge.

The step is important. We all agree it's important. We still skip it. Every time, the reason is the same: too much friction for the value it delivers in the moment.

Instead of asking “why aren’t people following the process?”, ask “what’s making this process hard to follow?”

If you've run into something similar (a process that kept getting skipped until you found a way to reduce the friction) I'd love to hear about it in the comments.

That’s a fine goal. But it misses the bigger opportunity.

The real value isn’t answers. It’s visibility.

Every question a user asks reveals something:

• A confusing explanation

• A missing example

• A gap in the product itself

When teams add an AI assistant to their docs, they don’t just get faster support. They get a window into exactly where users are struggling.

When your AI can’t answer, that’s not a failure. It’s a signal.

That shift changes everything.

From static reference to feedback loop

Traditional documentation is a one-way broadcast. You write it, publish it, and hope it helps.

But when you track what users actually ask, documentation becomes a conversation. You see patterns. You spot the same confusion appearing again and again. You find the blind spots your team never noticed because you’re too close to the product.

Suddenly, your docs aren’t just supporting users. They’re teaching you what to fix.

The insight you’re leaving on the table

If your documentation only answers questions, you’re capturing maybe 10% of its potential value.

The other 90%? It’s in the questions themselves.

• Which features confuse people most?

• Where do users get stuck in onboarding?

• What terminology doesn’t land?

With an AI assistant, that data exists. Most teams just aren’t collecting it.

A good AI assistant surfaces which questions your docs can’t answer, ranked by frequency.

From insight to action

Knowing where the gaps are is only half the problem. Fixing them is where most teams stall.

The best tools help you close those gaps with clear guidance on where to start.

Start here

Next time you look at your docs, don’t ask “did we cover everything?”

Ask “what are users still asking, and why?”

The answer will show you exactly what needs to change.

“Hey, can you review this?”

Two days later, I had 10 comments. Four of them were about commas. Two more about whether colons introducing lists should be bold or not. Another spent three paragraphs debating whether “setup” should be one word or two.

Not a single comment mentioned that step 4 was completely broken. The command I told users to run didn’t actually work.

Why vague requests produce surface feedback

When you ask “can you review this?” without direction, people default to whatever stands out first.

Typos. Formatting. Word choice. Easy to spot, easy to comment on.

“Review this” could mean:

• Check if the logic makes sense

• Verify the code runs

• Look for security issues

• Test the user flow

• Proofread for typos

• Check if the tone is right

Without knowing what you need, people pick the easiest path.

The cost of unfocused reviews

You spend hours sorting through comments that don’t help. Reviewers spend energy on things you’re going to ignore or fix a later phase. The real issues never get caught.

I’ve seen teams spend three review cycles debating admonition colors while nobody notices the registration flow is broken. Product and docs ships. Users can’t sign up. “But we had five people review it.”

Yeah, they reviewed the wrong things.

How to ask for useful feedback

Tell exactly what to look at:

• “Does the getting started flow make sense?”

• “Are the code examples working on your end?”

• “Is the reference documentation for this function complete?”

And tell them what to ignore:

• “Don’t test authentication, QA is handling that”

• “Ignore formatting, I’ll run the linter”

• “Skip the intro section, it’s placeholder text”

• “Don’t worry about performance yet, that’s next sprint”

If you’re the one being asked to review

This works the other way too. Someone sends you “can you review this?” with no context.

Ask “What kind of feedback are you looking for?”

Takes 30 seconds. Saves both of you from a useless review cycle.

See something else? Mention it. But stay focused.

If you spot a bug or broken link while reviewing, point it out. If something’s unclear, say so. Of course, leave things better than you found them. This isn’t about ignoring obvious problems.

But keep the ultimate goal in mind. If someone asked you to review the tutorial flow and you spend 15 minutes looking for oxford commas, you’ve lost the plot. Don’t forget what they actually needed.

Just don’t forget what they actually needed.

How to make this normal

Next time you ask for a review, add one specific question. When someone asks you to review something, ask what they’re looking for.

You can create simple templates:

Review Request:

• What changed: brief summary

• What to focus on: specific concerns

• What to skip: things handled elsewhere

• Questions: what you’re unsure about

What this comes down to

Everybody wants to help. But without direction, it’s easy to loose focus on whatever jumps out first. And that’s almost never what you need.

If you want useful feedback, ask useful questions.

Be specific. Point toward what matters. Call out what to ignore.

You’ll get faster reviews, better feedback, and fewer arguments about commas.

Most documentation disasters come from organic growth. One team adds a page here, another over there. Nobody’s thinking about the whole. Structure only becomes a concern after hundreds of pages exist, when nothing is easy to find or truly useful.

The pattern is always the same: content first, then more content, then nobody can find anything. Then someone finally asks “should we reorganize this?” when reorganization means touching 300 pages and breaking every bookmark and search result.

Strategy → Structure → Content.

Not Content → More Content → Chaos.

What documentation strategy actually means

Strategy doesn’t mean a 40-page plan. It means answering a few critical questions before anyone writes a word:

Who are you writing for?

Not “developers.” That’s too vague. Define specific personas in specific situations. A developer integrating your API under deadline pressure needs different content than an enterprise architect evaluating your security model. Pick a primary audience. Acknowledge the others exist, but know who you’re optimizing for.

What job is each piece of documentation doing?

Documentation serves different purposes:

• Teaching a concept through example (tutorial)

• Solving a specific problem (how-to guide)

• Looking up syntax (reference)

• Understanding why something works this way (explanation)

These aren’t interchangeable. A reference doc won’t teach a beginner. A tutorial won’t help someone troubleshooting at 2am. Your strategy needs to decide which jobs matter most for your product and your users, then commit to doing those jobs well.

What’s the minimum content that unlocks the product?

You don’t need comprehensive API docs on day one. You don’t need advanced guides for features nobody’s using yet. You need whatever removes the biggest obstacle between a new user and their first success.

Everything else can wait. The question isn’t “what should we document eventually?” It’s “what’s blocking someone right now?”

What should each page cover?

Once you know which content types you need, define what makes each one complete. Tutorials should list prerequisites up front and guide users to a working example. API references need request examples, response formats, and error codes. How-to guides should state the outcome at the start and show the steps clearly. Explanations need context on why things work the way they do.

This is where templates help. They ensure consistency and completeness across your documentation. But templates only work after you’ve defined your strategy. Without clarity on what each page type requires, templates just give you consistently incomplete content that raises more questions than it answers.

The early content trap

In practice, you can ship some early content while doing this thinking. In fact, you should. But those quick wins should not define the strategy.

You've seen this before: you ship a tutorial because five users asked for it. Then a troubleshooting doc because support is overwhelmed. Then someone adds an FAQ. Then best practices. Each one makes sense in isolation. None of them were planned together. Six months later, you have 50 pages organized by the order they were written, not the order anyone needs them.

A strategic early win looks different. You know it’s one piece of a larger system. You know where it lives. You know what comes before and after it. Even if you haven’t written those other pieces yet, you’ve left room for them.

This is the difference between “we need a tutorial, someone write it” and “we need a tutorial that assumes zero prior knowledge, gets someone to a working integration in under 10 minutes, and hands them off to either the API reference or advanced tutorials depending on what they want to do next.”

Same deliverable. Completely different foundation.

What changes when you do this right

Writing gets faster. You’re not reinventing structure with every new page or second-guessing where something belongs.

Maintenance gets easier. When something changes in your product, you know exactly which docs need updates because you understand how the pieces relate.

Users find what they need. Not because your search is amazing, but because the structure matches how they think about problems.

Teams can contribute confidently. A new PM can add content because the system makes it obvious where things go.

None of this happens if you start with content and hope structure emerges. Structure rarely emerges on its own. Chaos does.

How to start

You don’t need perfect answers to those strategy questions. You need good enough answers and permission to revisit them.

Spend a week, maybe two. Talk to five users about what they’re trying to do. Look at your support tickets and see what’s actually confusing people. Sketch out the 10-15 core topics your documentation needs to cover and how they connect.

What works for me: start by mapping navigation flows for each persona. Where does a junior developer enter your documentation? What’s their path to first success? What do they need next? Where does a senior architect start, and what questions do they need answered to evaluate your product? Sketch these journeys. Where each audience type enters, what they read, where they go when they’re stuck.

Example navigation flows for two different personas. Use it to identify key content and paths.

These flows reveal what content you actually need, not just what seems logical to document. Also, they will help you define your structure.

Structure isn’t about folders. It’s about reflecting how users think about your product. If your users think in terms of workflows, organize by workflow. If they think in terms of features, organize by feature. Don’t organize by how your engineering team structured the codebase. Nobody outside your company thinks that way.

Then create a grid in a spreadsheet. One row per page or section you think you’ll need based on the previous exercise.

Columns for:

• Module / page title

• Persona: who is this for?

• Content type: tutorial, reference, how-to, explanation

• Priority: must-have for launch, nice-to-have, future

• Description: Brief description of what the page covers, required subheadings.

The grid forces you to think about the whole system at once instead of page by page. You’ll see gaps (”wait, we have five reference docs but no how-tos?”) and redundancies (”these three pages are basically the same thing”).

What I’ve found useful: identify your action-oriented content first (tutorials and how-to guides), then build supporting reference and explanation docs around them. These supporting docs can be referenced from multiple guides, so you’re not repeating the same API details or concepts in every tutorial. This keeps guides focused on the task while providing depth for users who need it.

Together, the flows and the grid become your roadmap. When someone asks “should we document X?” you can see where it fits, or doesn’t. When priorities shift, you adjust the plan, not 47 individual pages.

Then start writing. But you’re writing into a framework now, not into a void.

Strategy doesn’t slow you down. It’s what makes fast sustainable.

Planning your documentation strategy?

If you’re staring at documentation chaos or launching something new and want to avoid building problems into your foundation, we can help.

We work with teams to design content architecture that scales: mapping personas, defining structure, and creating the roadmap before anyone writes a word.

2025 was full of growth and learning.

At TechDocs Studio, we helped clients build scalable documentation infrastructure, automate workflows, and improve their docs, seeing firsthand how targeted improvements transform daily work for teams.

We also reinvested in our own products, including Biel.ai and PushFeedback, tackling common documentation challenges our customers face. Watching these tools start to make a real difference has been incredibly rewarding.

This year reinforced that growth is not just about new contracts, but about trust, collaboration, and celebrating small wins together.

I’m grateful to everyone who made this year possible:

• 𝗖𝗼𝗿𝗲 𝘁𝗲𝗮𝗺: Àlex, Raúl. Thank you for bringing skill, dedication, joy to every project.

• 𝗡𝗲𝘄 𝗰𝗹𝗶𝗲𝗻𝘁𝘀: Kristy-Leigh, Matthew, Rustom, Alex, Jimmy. Thank you for trusting us.

• 𝗟𝗼𝗻𝗴-𝘁𝗲𝗿𝗺 𝘀𝘂𝗽𝗽𝗼𝗿𝘁𝗲𝗿𝘀: Anna, Tzach, Elena, Jorge, Alberto, Rich, Nate. Thank you for your continued trust and support over the years.

• 𝗘𝗻𝗮𝗯𝗹𝗲𝗿𝘀: Pablo, Juan Luis. Thank you for connecting us with exceptional people.

• 𝗙𝗲𝗲𝗱𝗯𝗮𝗰𝗸 𝗰𝗵𝗮𝗺𝗽𝗶𝗼𝗻𝘀: Lars, Tim, Harshil, Seonaid, Elena K. Your feedback improved our tools for everyone.

• 𝗜𝗻𝘀𝗽𝗶𝗿𝗮𝘁𝗶𝗼𝗻: Ivan, Sabela, Cristina, Aleix, Xavi. I’ve learned a ton from all of you.

• 𝗢𝗽𝗲𝗿𝗮𝘁𝗶𝗼𝗻𝗮𝗹 𝘀𝘂𝗽𝗽𝗼𝗿𝘁: Davinia for keeping our finances in order, Gonzalo for managing customer relations.

...and to everyone we worked with behind the scenes, thank you for your support.

If you’re curious about what we built in 2025, you can explore our projects at techdocs.studio/work:

Aligning Coiled’s documentation theme with their brand identity

Following the successful launch of Coiled’s new website, we developed a custom Sphinx documentation theme that aligned with their updated brand identity, ensuring a cohesive user experience across docs.coiled.io and the main marketing site.

Read case study

Custom Sphinx theme development for FiftyOne documentation

We upgraded FiftyOne’s documentation from Sphinx 3 to the latest version, created a custom theme aligned with the new Voxel51 brand guidelines, and implemented advanced search capabilities with PushFeedback integration for continuous improvement.

Read case study

From Webflow to Next.js: Creating a developer-first website for Coiled

Migrated Coiled’s website from Webflow to a modern, component-based static site generator, empowering their team to manage content like they manage code.

Read case study

Automating API reference generation from OpenAPI and GraphQL schemas

Migrated MONEI’s docs to Docusaurus v3, implemented cross-site search, and autogenerated consistent API documentation from OpenAPI and GraphQL contracts.

Read case study

Unifying Blue Robotics’ documentation with a custom Sphinx theme

Developed a custom Sphinx theme with Material design aesthetics, implemented multi-language support, and set up a custom CI/CD pipeline for version management across their documentation ecosystem.

Read case study

Here’s to 2026: more learning, more impact, and helping teams build documentation that scales!

— David

We suggested adding a feedback widget. A small button: “Was this helpful?”

One week later, 12 users had left feedback. Not one was positive.

“Step 3 is missing.” “This example doesn’t work.” “I’ve read this four times and I’m still confused.” “Where’s the actual code?”

Those 5 minutes weren’t engagement. They were users scrolling up and down, re-reading, trying to figure out what was missing.

Good metrics, bad experience

The analytics showed what users did. The feedback showed why.

The team fixed everything in 48 hours. Added the missing step. Replaced the broken example. Clarified the confusing parts. Added code snippets where users expected them.

Session time dropped to 3 minutes. But now users were finishing successfully instead of giving up confused.

You need both

Quantitative data shows behavior. Qualitative data explains it.

Your analytics will tell you users spent 5 minutes on a page. Your users will tell you they spent 5 minutes confused.

One number can mean two completely different things.

Formatting notes sit beside broken code examples. The team spends 30 minutes debating comma placement while critical bugs slip through.

To fix this, I introduced a tagging system for review comments.

The problem: No shared language for priority

Reviewers leave comments without indicating priority. Contributors treat every comment as blocking.

The breakdown happens in three ways:

• Contributors waste time on non-issues: Someone suggests “maybe use a bullet list here instead of numbered?” The contributor spends an hour reformatting. Turns out it was just a passing thought.

• Critical issues get missed: Seventeen comments deep, one mentions the API endpoint is wrong. The PR merges. Users hit a broken example. Support tickets pile up.

• Review cycles multiply: Contributor fixes what they think matters. Reviewer expected different fixes. Another round. The two-day PR becomes a two-week ordeal.

The root cause: no shared language for priority. And even when something clearly needs fixing, the comment doesn’t say what kind of fix. Is it a code bug? A clarity problem? A style preference? Contributors guess, guess wrong, and the cycle repeats.

The fix: severity + category

A two-part tag for every review comment.

Severity tells contributors the importance of the fix:

• blocker: Must fix before merge. Breaks functionality or blocks critical understanding.

• improvement: Should fix before publication. Improves clarity, structure, or usability.

• optional: Personal preferences, stylistic suggestions, future ideas.

Category

Category tells contributors what kind of fix at first glance:

• clarity: Confusing or incomplete explanations.

• bug Incorrect code, wrong output, factual errors.

• organization: Issues with flow, structure, content order.

• style: Wording, formatting, tone.

• consistency: Conflicts with other sections, terminology, or examples.

Add others if your team needs them. Keep the list short.

Some examples

(improvement, clarity): Step 3 assumes readers know what environment variables are. Add a one-line explanation or link to prerequisites.

(blocker, bug): This code example throws an error. Missing import statement at the top.

(improvement, organization): Move the troubleshooting section before advanced configuration. Users hit errors at the basic level first.

(optional, clarity): This explanation works, but the second paragraph repeats the first. Consider cutting it.

What changes after tagging

• Contributors know what to fix immediately. Scan for blocker tags. Fix those first. Everything else waits.

• Contributors feel less overwhelmed: Seventeen comments looks terrifying. Two blockers, eight improvements, seven optional? Manageable.

• Debates end faster: No more “should we merge this or wait?” If there are no blockers, merge. Track improvements for follow-up.

• Review culture improves: New reviewers see the pattern. They start tagging their comments. The team develops shared expectations.

What this doesn’t fix

• Tagging doesn’t fix slow reviewers. If your team takes three days to look at PRs, tags won’t help. That’s a process problem.

• Tagging doesn’t fix bad feedback. “(blocker, style): I don’t like this wording” is still bad feedback. Add context. Explain why.

• Tagging doesn’t replace good judgment. A reviewer still needs to know what’s actually a blocker vs. a preference.

How to start

Don’t overthink it. Start tagging your review comments this week.

Put the tag at the start of each comment. Make it visible.

Within two weeks, you’ll see fewer debates about what blocks merge. Contributors will know what to fix first. Reviews will move faster. Others will copy you.

The system works because it forces one decision upfront: is this blocking or not?

Make that decision explicit. Stop wasting time guessing.

The waitlist just opened: https://tally.so/r/RGxzzJ

Join for early access and help decide what features to build first.

What’s the most annoying part of your screenshot workflow? 👇

Then things drift apart.

The problem

Engineer adds a new configuration property:

export interface DatabaseConfig {
  isolationLevel?: IsolationLevel;
  enableSsl?: boolean; 
  retryAttempts?: number; // New property
}

Deploys it. Forgets to update the docs.

Three weeks later, a user reads the documentation. No mention of retry_attempts. They assume the feature doesn’t exist. Or worse, they assume the default behavior and it breaks their setup.

Support ticket arrives.

This pattern repeats with every config change. Properties added, docs lag behind. Properties deprecated, docs still show them. Default values change, docs show old defaults. The same thing happens with API parameters, CLI flags, environment variables, error codes. Anywhere docs describe what exists in code.

The root cause is simple: code and docs live in two places with different owners. Code changes frequently, gets reviewed and tested, and is always correct. Docs update manually, have no automated checks, and drift out of sync. Two systems that don’t talk to each other.

The solution: define once, generate the rest

There are two approaches.

Approach 1: Define a schema

Write a machine-readable schema as your source of truth. From it, generate code, docs, and everything else.

properties:

  - name: isolation_level
    required: true
    type: string
    enum: [READ_UNCOMMITTED, READ_COMMITTED, REPEATABLE_READ, SERIALIZABLE]
    example: READ_UNCOMMITTED
    description: Transaction isolation level

  - name: enable_ssl
    required: false
    type: boolean
    default: true
    description: Enable SSL encryption for database connections

  - name: retry_attempts
    required: false
    type: integer
    default: 3
    description: Number of times to retry failed connections

From this schema, generate server code, client SDKs, API documentation, examples, interactive API explorers. The schema is your contract. Everything else derives from it.

An additional benefit: by decoupling the schema from actual code, you facilitate conversations about the contract. Teams can review and agree on the API structure before implementation.

This approach works best when open standards exist for your technology: OpenAPI for REST APIs, GraphQL schemas, Protocol Buffers for gRPC, JSON Schema, AsyncAPI for event-driven architectures.

If no standard exists, you need to build the generators yourself, which could be a stopper if resources are limited.

Approach 2: Annotate your code

Add documentation as annotations directly in your code. Generate docs from it.

Example: TypeScript with JSDoc

export interface DatabaseConfig {

  /** Transaction isolation level
   * @required
   * @example ‘READ_COMMITTED’
   */
  isolationLevel?: IsolationLevel;

  /** Enable SSL encryption for database connections
   * @default true
   */
  enableSsl?: boolean;

  /** Number of times to retry failed connections
   * @default 3
   */
  retryAttempts: number;
}

From annotated code, generate the intermediate schema and the reference documentation.

Check if your programming language already has a standard: JavaDoc for Java, Google-style docstrings for Python, JSDoc for TypeScript/JavaScript. If no convention exists, define one. Document it. Stay consistent.

This approach is simpler than defining schemas, especially when you don’t have code generators available.

How to implement it

Document your source of truth. Write your schema file or add annotations to your code. Include descriptions, types, defaults, constraints.

Be thorough. This is the single source. Everything generates from it.

Choose your generator. Check if existing plugins work for your docs platform. Rendering OpenAPI in Docusaurus? There’s a plugin. Sphinx? Extension available.

But if you need something specific (TypeScript docs in Antora, for example), you’ll need a custom solution.

Key tip for annotations: Always generate an intermediate schema file (JSON or YAML) first. Don’t try parse code annotations directly in your docs build.

If you need a custom extension, that requires dev work. We can help with that.

Add to CI. On every commit: validate the source, regenerate documentation, fail the build if validation fails. On merge: deploy updated documentation automatically.

Start small

You don’t need to generate everything on day one.

Start with one configuration file. Generate just the reference table. Keep the rest manual. You can still have custom, hand-written guides alongside generated reference sections. Prove it works. Then expand.

A common objection

“But our configuration is complex. We need custom documentation.”

You can still have it. Generate the reference section (the part that can be autogenerated and reused in guides). Keep step-by-step guides manual.

The result

Engineer adds a new parameter. Commits the code. The docs update automatically.

Three weeks later, a user reads the documentation. The parameter is there.

No support ticket this time.

The content was technically correct but generic. Corporate tone that didn’t match our style. Examples that could have been for any product. Missing the context that makes ours different.

The AI wasn’t the problem. I was asking it to write documentation without giving it the information it needed.

What changed

We now keep a context file next to the codebase. Every time I prompt an AI agent, I feed this file in first.

Without context file:

“Welcome to Biel.ai! We’re excited to help you get started with our amazing AI chatbot platform.”

With context file:

“Biel.ai trains on your documentation to answer technical questions. This guide gets you from zero to a working chatbot in 10 minutes.”

The second version matches our voice. Uses our positioning. Focuses on the user.

Think of it like onboarding a new team member. You don’t say “go write docs” and expect great results. You explain the product, the users, the style, the constraints. Same with AI.

How to build one

Create a markdown file next to your codebase. Include:

1. Three sentences about your product

2. Who uses it and why (specific personas, not “developers”)

3. How you write, what to avoid, examples of good vs. bad. Paste 3-5 examples from your best docs (this is your voice)

4. Common use cases (actual workflows)

5. Technical details (stack, repo structure, naming conventions)

Commit it so it versions with your code. Update it as you find the AI producing results that miss the mark.

Use it for docs, marketing copy, support responses, tutorial content, release notes. Anywhere AI writes for your product.

More advanced: If your IDE supports rules (like Cursor’s .cursor/rules), add the context file there. It’ll feed into every prompt automatically.

An unexpected benefit

The context file started aligning the whole team. New engineers read it to understand positioning. Writers use it for consistency. Product managers reference it when describing features.

It became our single source of truth for “how we talk about this product.”

Worth the 30 minutes to set up.

Your content stays in their system. Your build process depends on their infrastructure. Your ability to move depends on their export tool, which barely works.

Docs-as-code flips this. Content is markdown in Git. You own the build process. You can host anywhere. Switch tools without losing content.

This isn’t anti-SaaS (there are docs-as-code friendly platforms out there). It’s about keeping ownership of your content so it doesn’t become leverage for someone else.

Your docs are a business asset. Treat them like one.

P.S. I’m back on Substack with a weekly note on technical writing, docs-as-code, and automation. If you’re not interested, you can unsubscribe anytime.

An AI chatbot lets them ask in plain language and get instant answers from your existing documentation.

Here’s how to add Biel.ai to your documentation. This guide uses Docusaurus, but Biel.ai works with any static site generator.

Disclaimer: I built Biel.ai. It’s a paid service with a free trial.

Prerequisites

• Docusaurus site (existing or new)

• Node.js v18 or higher

• A sitemap for your docs site

Step 1: Create a Biel.ai account

Go to app.biel.ai and sign up. Follow the on-screen instructions to create your account.

Step 2: Create your first chatbot project

1. In your Biel.ai dashboard, click on the Projects tab at the top.

2. Click Create project.

3. Enter your project details:

   • Project name: Choose a unique name.

   • Sources: Select Sitemap and enter the sitemap URL of your Docusaurus site (e.g., docs.domain.com/sitemap.xml).

4. Click Save.

You’ll receive a unique Project ID. Keep this handy, as you’ll need it later.

Step 3: Install the chatbot

1. Install the Docusaurus package:

npm install docusaurus-biel

2. Add Biel.ai to your docusaurus.config.js file:

plugins: [
    [
        ‘docusaurus-biel’,{
            project: ‘<PROJECT_ID>’,
            version: ‘latest’
        }
    ]
],

Replace <PROJECT_ID> with the Project ID you received earlier.

3. Run your site locally to ensure everything is working properly:

npm start

The chatbot appears as a floating widget. Users can ask questions and get answers from your docs:

Step 4 - Customize your chatbot (optional)

You can tweak the chatbot’s appearance and behavior in the docusaurus.config.js file. For example, you can change the button text or adjust the chatbot’s position:

plugins: [
    [
        ‘docusaurus-biel’,{
            enable: true,
            project: ‘<PROJECT_ID>’,
            bielButtonText: ‘ASK AI’,
            buttonPosition: ‘center-right’,
            modalPosition: ‘sidebar-right’,
            headerTitle: ‘Biel.ai chatbot’
            buttonStyle: ‘dark’,
        }
    ]
],

For advanced customization options, such as adding an initial message, suggested questions, or further editing the style, refer to Customization.

What the chatbot conversations tell you

The chatbot tracks what users ask. This reveals gaps in your documentation.

If 50 people ask about webhook authentication but you don’t have a guide, that’s a signal to write one. The questions show what’s missing or unclear.

The AI only works as well as your documentation. Well-structured, comprehensive docs lead to better answers. Gaps lead to poor responses.

It’s right there in the docs. But the user searched “automatic notifications” and got nothing. The docs call it webhooks. The user doesn’t.

Traditional search works when users know the right terms. They usually don’t. They describe what they’re trying to do, not what the feature is called. Your docs could be perfect and users would still miss them because they searched the wrong word.

What we built

Over the past year, we’ve been building Biel.ai to solve this. It’s an AI chatbot that sits on your docs site and answers questions from your existing content. Think of it as ChatGPT, but trained on your product documentation.

Users ask a question in plain language. Biel finds the answer in your docs and responds. No keyword matching. No browsing through a sidebar hoping to find the right page.

What surprised us

Biel didn’t just help users find answers. It showed us what was missing.

The conversation logs revealed gaps we didn’t know about. Topics we assumed were covered but weren’t. Workflows that lived in Slack but not in the docs. Basic things nobody had written down.

We went from guessing what users needed to seeing it in the data. The chatbot became a feedback tool as much as a support tool.

Who this is for

If you maintain documentation for a technical product and your users keep asking questions the docs already answer, Biel can help.

It works with any static site generator. Setup takes about 10 minutes.

Want to see it on your docs? Send me a message at david@techdocs.studio with your docs URL and I’ll set up a demo.