Many knowledge base projects start by uploading every PDF and web page into a vector store and wiring it to a chatbot. The demo usually goes well. The problems show up after launch:

  • Three versions of the same contract all come back, and the answer cites the one that is no longer in force.
  • A user types a document number, and vector search returns a few unrelated clauses that “mean something similar”.
  • Someone finds a conversion error but cannot tell which original and which page it came from, or what else the fix affects.
  • Business reviewers want to see what the knowledge base actually contains, and the only place to look is the vector store.

None of these are retrieval problems. They happen because the content itself is not managed as a product.

We treat the knowledge base like a library, or a dictionary: categories first, then books and their editions, a table of contents inside each book, and only then the actual passages. The chat model doesn’t need “the whole knowledge base”, only enough to answer the question. The structure exists to narrow the context level by level and hand the chat model accurate knowledge.

This post draws on a bilingual (English and Chinese) knowledge base project: about 1,800 Markdown documents and 800-plus Q&A entries, with long, clause-heavy PDFs as the main originals. It covers the content model, the publishing flow and the retrieval choices. How every source becomes Markdown and how it is chunked by heading is the subject of the next post.

1. Reference practices

The reference architectures from the major cloud and model vendors are clear about what has to happen before retrieval:

SourceMain practice
Azure Architecture Center, RAG solution seriesA preparation phase first: understand document types, strip headers, footers and watermarks, then chunk by document structure; hybrid queries and reranking at retrieval time
Amazon Bedrock Knowledge BasesParsing, chunking and metadata filtering are separate settings; a foundation-model parser for documents with tables and figures
Google Document AI Layout ParserDetect layout elements first, then create chunks whose text all comes from one layout element
Anthropic, Contextual RetrievalCombine embeddings with BM25, then rerank; retrieval failures drop clearly compared with embeddings alone
Write the Docs, Docs as CodeDocs in plain-text markup, under version control, through code review and automated tests

What they share: parsing, chunking and indexing are steps you can inspect separately. Sections 2–6 are our design. Section 7 compares it with ready-made platforms (RAGFlow, WeKnora, Azure AI Search) and explains why we didn’t adopt them wholesale.

2. Like a library: narrow the context level by level

LevelIn a libraryIn the knowledge base
CategoryShelf sectionCatalog: customer segment › product line › product
DocumentAn edition of a bookOne Markdown file with type, language, version, status
SectionThe book’s table of contentsThe Markdown heading tree and heading path
ChunkA passage in a sectionThe body under one heading
Q&AA dictionary entryOne Q&A entry, belonging to one level

The context the chat model receives is narrowed down through these levels, in one of two ways, each with its own interface:

  • You know the shelf and the book: exact fetch by metadata. The chatbot already knows the product and language and fetches “the full current terms for this product” and “all Q&A for this product”, with no wrong version and no half a clause.
  • You only have a question: look it up in the index. Search indexes chunks and returns them with heading path and page, grouped by document; when more context is needed, walk up the heading path to fetch the whole section or document.

The context changes as the conversation goes on

The context isn’t fixed up front. The further the conversation goes, the more is known, and the narrower the scope can get:

Conversation stageWhat is knownWhat goes into the contextStructure it relies on
StartOnly a questionGeneral rules, an overview of each product line, general Q&AThe top level of the catalog; this part is fixed and can sit in the prompt cache
A product line comes upThe product lineIntroductions and Q&A for the products in that lineCatalog: product line › product
A specific product is settledProduct, language, maybe a signing dateThe full terms of the matching version, or the relevant sectionsDocument version and status
A specific clause is asked aboutThe exact questionRetrieved chunks, plus the whole section via the heading path when neededHeading tree and heading path
The topic changesA new product or questionReplace the previous level’s content instead of piling it upAs above

Today our chatbot detects the product line by keywords and would rather load too much than miss something; later the model could decide through tool calls which level to fetch next. Whoever decides, the precondition is the same: every level has a stable, accurate structure to locate content by. Without it, every turn can only search the whole library or stuff everything into the context.

A mistake at any level gives the chat model the wrong context: a wrong category finds the wrong book, a wrong version cites terms no longer in force, a wrong heading tree cuts half a clause. So the content model (section 3) and the heading tree (next post) matter as much as retrieval. When the caller already knows which book it needs, skip retrieval.

3. Content model: one source, one Markdown document

Whether it started as a web page, a PDF or a manually supplied file, everything becomes “Markdown with metadata”:

  1. One source, one document. Metadata covers type, language, place in the hierarchy (customer segment, product line, product), version, status (current or historical) and source address.
  2. Traceable to the original. Each Markdown file records the original’s sha256, and text converted from PDF keeps page markers, so any passage leads back to a page.
  3. Old versions are kept. A contract signed under older terms still needs those terms; they are marked historical, and search covers current versions by default.
  4. Bilingual files are split by language. Conversion tags each block with its language, then splits the file into two documents; line-by-line bilingual covers and tables go to both.

Q&A is not a document and has its own model: one question and answer is one record and belongs to exactly one level (a product, a product line or a customer segment); scraped and hand-written Q&A share the model and are told apart by a source field; the FAQ section is removed from product pages so each Q&A is stored once. The chatbot can then fetch “all Q&A for this product” in one call, and the search page treats each Q&A as one chunk.

4. Store and publish with Git

Bodies and metadata live in Git, not in a database.

Overall structure: sources pass through adapters into a Git-managed Markdown knowledge base, with originals stored by hash outside the repository; after merge, it serves both exact fetch and the search index

  • Git holds only Markdown bodies, Q&A and a little YAML metadata. PDFs, page originals and OCR intermediates (about 1.2 GB) live in an original store outside the repository, keyed by sha256 and append-only.
  • Every sync is a branch and a pull request. The website sync and internal-system sync are just source adapters that produce pull requests instead of writing to a database. Reviewers read the diff of the converted text, and merging to main publishes it.
  • People write only their decisions. Each level has one hand-maintained file recording what no source provides, such as which product a file belongs to. Sources merge into one effective catalog, and conflicts go into a report.
  • The service reads the repository at startup and builds an in-memory index that serves every exact-fetch endpoint.

Git provides versions, history and audit, so no revision tables are needed, and when a conversion rule changes the pull request shows exactly which lines of which documents changed. It fits thousands of documents, batch updates and no per-user access control. Beyond that, metadata belongs in a database, but the body should stay diffable and reviewable.

5. Search: pair hybrid with a reranker

In the index, one record is one chunk, with document-level fields (type, product, language, status) copied onto every chunk for filtering and facets. Three more points:

  • The embedding input is “document title + heading path + chunk body”, not the body alone.
  • Exact strings such as document numbers go into a separate field with a keyword analyzer, matched whole and weighted higher.
  • Each chunk carries its page and position in the source, so the preview jumps straight to it.

We compared methods on the same 48 questions (English, Traditional Chinese, colloquial Simplified Chinese, number lookups):

MethodR@5MRR@10
Keyword (BM25)77–79%0.68–0.71
Vector96%0.918
Hybrid (RRF, no reranking)94–96%0.846–0.878
Hybrid + managed semantic ranker100%0.936
Hybrid + standalone reranking model100%0.905–0.911
  1. Hybrid without reranking did worse than vector alone. The keyword side ranked poorly, and RRF pushed good vector results down; a reranker recovered the loss and then beat vector alone.
  2. Keyword search cannot be dropped. Only it finds document numbers; vector search misses them entirely. Azure’s hybrid search docs say the same about product codes, jargon, dates and names.
  3. No conversion across languages or Chinese scripts. The knowledge base is in Traditional Chinese and users often ask in Simplified. Keyword matching fails, but embeddings map Simplified questions to Traditional clauses and English questions to Chinese passages. So the search page defaults to hybrid plus reranking.
  4. The search page does not generate answers. The managed service’s “answer” is a sentence extracted from the source, shown only when the query looks like a question and scores high enough; for number lookups it is usually irrelevant.

6. The evaluation set decides, and has blind spots

The evaluation set decides the retrieval method:

  • Questions follow how people actually ask: formal, casual, Simplified and Traditional, mixed languages, numbers.
  • Each question marks the documents that count as correct, narrowed to a section or a specific Q&A where needed.
  • Every method gets the same questions and scoring rules, measured with R@1, R@5, R@10 and MRR.
  • No tuning on a small set. With 48 questions, one question is 2 percentage points. Keyword weights of 1.0, 0.7, 0.5 and 0.3 and RRF k of 60 and 20 all stayed within 1–2 questions, so we kept default equal-weight RRF.

Ranking metrics only ask whether the right document is near the top, not whether a chunk is complete or its heading path is right. The next post shows many chunking improvements that barely move the ranking but directly affect how results display and how well clauses are cited. The next step is 150–200 real questions and an answer-level metric, such as “does the top chunk contain the complete answering clause”.

7. Why we ended up with this approach

RAGFlow, WeKnora and Azure AI Search turn parsing, chunking and retrieval into product features. We borrowed a lot from them but did not adopt any of them wholesale:

OptionWhat it does wellWhy we didn’t use it directly
RAGFlowParsing templates per document type (Q&A, manual, laws and more), one Q&A pair per chunk; chunk results visualized and editable against the source; a retrieval testing page for keyword weight and rerankerManual fixes land on chunks, so re-parsing needs extra care to reconcile them with the body; review is chunk by chunk, not a whole-document diff in a pull request; the focus is retrieval and chat, while our main caller wants whole documents by metadata
WeKnoraAutomatic sync from Confluence, Notion, Feishu and others; a separate FAQ library; chunk edits with revision history, diff and rollback; parent-child chunks, hybrid retrievalSame as above: the source of truth is the chunk, and exact fetch by metadata is not its focus
Azure AI Search built-in indexers and skillsetsAutomatic parsing, chunking and vectorization; hybrid search, semantic ranking, extracted answersThe built-in layout skill runs layout OCR again, exactly where we had trouble with two-column pages and split tables; skills only run inside an indexer, with no standalone API; vectorization couldn’t reach our own model gateway
Own content layer + Azure AI Search for retrieval only (chosen)Reviewable, traceable, versioned content; conversion and chunking rules can be iterated and rerun in fullWe maintain conversion, chunking and push code ourselves, and there is no ready-made chunk editor

The key difference is which layer holds the source of truth. On these platforms, manual fixes happen on chunks, so chunks become the source of truth. We fix the Markdown body; chunks are derived, chunk IDs come from content, every chunk can be regenerated when the rules change, and embeddings are recomputed only where content changed. Retrieval goes to Azure AI Search through its push API, using only hybrid search and semantic ranking.

To be clear: we did not benchmark RAGFlow or WeKnora on our data, so the table reflects design trade-offs. The problems with Azure’s built-in skills are ones we actually hit in the POC.

References