In most companies, the knowledge already exists: in project folders and wikis, in e-mails and ticket systems, in contracts, manuals and in the heads of experienced colleagues. It just is not available. Anyone who needs an answer searches several systems, asks around or reinvents the wheel. And when knowledge carriers leave the company, their knowledge often leaves with them. Artificial intelligence can change exactly that: it makes scattered corporate knowledge available like an expert help desk – in seconds, with source references.
Why not simply ask a language model?
Large language models write impressively fluent text – but they do not know your company. Their knowledge comes from public training data with a fixed cut-off date. When asked about internal topics, they either cannot answer or, worse, respond with convincing-sounding inventions – so-called hallucinations. Three essential things are missing for enterprise use: access to your documents, up-to-date information and verifiability.
The solution is retrieval-augmented generation (RAG): the language model does not answer from memory, but receives the relevant passages from your own documents for every question – and cites the sources the answer is based on. This turns a language talent into a reliable expert resource.
How RAG works – explained simply
A RAG system works in two steps. First, the preparation: your documents are split into meaningful sections (chunking) and translated into a mathematical representation (embeddings) that captures the meaning of a text – not just its words. These representations are stored in a vector database.
Then, the answering: when someone asks a question, the system finds the sections that match best in content – even if they use completely different words than the question. The language model receives these passages as context and formulates an answer from them, backed sentence by sentence by the original sources. Anyone who doubts the answer clicks the source and reads for themselves.
Stage 1: Naive RAG – the quick start
The basic form – one search step, one answer – is deliberately simple, and that is exactly what makes it the ideal entry point: quick to set up, easy to understand, immediately useful. For clearly phrased questions on a well-maintained document base ("How do I apply for…?", "What does contract X say about…?"), naive RAG already delivers reliable results and makes the benefit tangible within the company.
The approach shows its limits with ambiguous questions, with facts scattered across many documents, and wherever relationships matter: who is connected to whom, what belongs to which project, which rule replaces which older version? A single search step finds text passages – but no relationships. And purely semantic search has an often underestimated gap: exact identifiers such as article numbers or error codes. Why that is – and how to solve it – becomes clear when looking at the two decisive levers of a RAG system.
Lever 1: Chunking – how documents are split
The most inconspicuous decision with the greatest impact: how are documents split into sections? What is lost here cannot be recovered by even the best search. Sections that are too large dilute the results with irrelevant content; sections that are too small tear statements out of context – a table without its heading, an exception without its rule. Several approaches have become established in practice:
- Fixed-size blocks: The text is cut after a fixed number of characters. Simple and fast – but the cut runs right through sentences, tables and related paragraphs.
- Fixed with overlap: Neighboring blocks share a margin so that nothing is lost at the cut boundaries. The proven standard for getting started, well suited to homogeneous prose.
- Structure-aware chunking: Cutting follows the document structure – headings, paragraphs, tables, lists. This requires clean parsing of the formats (PDF, Office, wiki), but pays off significantly for structured documents such as contracts or manuals.
- Semantic chunking: The most advanced variant: topic changes are detected by meaning – the cut is made where a new thought begins, not where a character counter dictates it.
In addition, we enrich sections with context – document title, chapter, validity – so that every passage remains understandable and correctly attributable on its own. And: there is no universally correct chunking. Contracts, tickets and technical manuals need different strategies – which one is right is determined by evaluation on your real document base.
Lever 2: Retrieval – finding what is actually meant
Semantic search via embeddings is RAG's great strength: it finds passages by meaning, even if they use completely different words than the question. But it has a systematic weakness – and it shows up in precisely the most precise questions. Anyone searching for article number "4711-B", an error code or a specific clause often gets no results from purely semantic search – or, worse, similar-looking but wrong ones. To an embedding, two article numbers are close together; to your purchasing department, they are worlds apart. Classic RAG fails on exactly these queries.
A production-grade RAG system therefore combines several techniques:
- Hybrid search: Semantic search and classic full-text search run in parallel and the results are fused. Meaning and exact strings both count – article numbers match again.
- Re-ranking: A second, more precise scoring stage examines the candidates in detail and moves the truly relevant ones to the top.
- Metadata filters: Validity period, document type, department, permissions – the search only considers what is actually eligible.
- Query understanding: The question is prepared before the search – rephrased, sharpened or split into sub-questions – so that colloquial requests become precise queries.
- HyDE (Hypothetical Document Embeddings): Instead of searching with the question itself, the AI first drafts a hypothetical answer – and its embedding is used for the search. A fully worded answer resembles the actual document passages far more closely than a short question, surfacing matches that pure question search would miss.
Stage 2: Advanced RAG and GraphRAG – precision and context
In the second stage, these levers come together: structure-aware or semantic chunking, hybrid search, re-ranking and metadata filters turn the simple question-answer system into a precise search across heterogeneous repositories – from contracts to ticket systems.
GraphRAG goes one step further: a knowledge graph is built from your documents – a network of entities (customers, projects, products, contracts, people) and their relationships. This makes questions answerable that go beyond individual documents: "Which projects depend on supplier X?" or "Which contracts affect location Y?" – questions where pure text search fails because the answer is not written in any single document.
Stage 3: Agentic RAG – the AI does the research
The highest stage fundamentally changes the role of the AI: instead of answering a question with a single search pass, an AI agent plans the research. It breaks complex questions into sub-questions, searches different sources step by step, assesses the quality of the results, queries the data base again where needed – and finally verifies its own answer against the evidence it found.
This pays off wherever people do laborious research today: preparing audits, analyzing contract portfolios, investigating technical incidents across years of tickets and documentation. Agentic RAG does not just answer questions – it completes research assignments.
The RAG landscape at a glance
Beyond the three stages, research and practice have produced a whole family of RAG architectures. Roughly, they fall into query transformation (HyDE), intelligent control (CRAG, Self-RAG, Adaptive RAG), specialized indexing (RAPTOR, multimodal), relationship knowledge (GraphRAG, LightRAG) and agentic research (agentic and multi-agent RAG). The most important ones compared:
| Architecture | Complexity | Answer quality | Cost & latency | Use case |
|---|---|---|---|---|
| Naive RAG one search step, one answer |
low | good for clearly phrased questions on a well-maintained base | low | FAQ, policies, quick pilot to prove the value |
| Advanced RAG hybrid search + re-ranking + query transformation |
medium | high – including technical terms, article numbers and codes | moderate | production standard for heterogeneous document bases |
| HyDE Hypothetical Document Embeddings: the AI first drafts a hypothetical answer and searches with its embedding |
low–medium | high for vague, tersely phrased questions | moderate (one additional generation step) | colloquial questions far from the documents' wording; quality gain without rebuilding the search |
| Modular RAG a construction kit with interchangeable modules for indexing, search and generation |
medium–high | high, each module can be optimized individually | moderate | several knowledge sources and requirements under one roof; a base that can grow |
| CRAG Corrective RAG: evaluates retrieved passages and corrects weak results before answering |
medium | high, robust on patchy document bases | moderate (additional verification steps) | repositories of varying quality; when wrong answers are expensive |
| Self-RAG the model decides for itself whether to retrieve, and critiques its own answer |
medium–high | high, fewer unnecessary searches | moderate | mixed workloads of knowledge and general questions |
| Adaptive RAG chooses the appropriate path per question: direct, simple or multi-step |
medium–high | high for strongly mixed question types | optimized – simple questions stay cheap | large user bases with very diverse questions; cost control |
| GraphRAG / LightRAG knowledge graph of entities and relationships; LightRAG as the lightweight variant |
high (LightRAG lower) | very high for relationship and overview questions | initial effort high (LightRAG cheaper), queries moderate | relationship-rich domains: projects, contracts, supply chains |
| RAPTOR hierarchical summary trees: answers at detail and overview level |
medium–high | very high for questions across entire repositories | index building costly, queries moderate | large text repositories with detail and overview questions (reports, studies) |
| Multimodal RAG e.g. ColPali: searches documents as images – layout, tables, diagrams, scans |
medium–high | very high for visually rich documents | higher indexing and storage costs | technical drawings, data sheets, scanned archives, table-heavy PDFs |
| Agentic RAG single agent (ReAct): plans the research in loops – think, search, assess |
high | highest for complex research – with verified answers | high; answers in seconds to minutes | research assignments, audit preparation, incident analysis |
| Multi-agent RAG specialized agents for search, verification and synthesis work in parallel with divided roles |
very high | highest for broad, multi-source research with cross-checking | highest | company-wide research across many systems; critical analyses |
The architectures are not mutually exclusive – they build on each other and can be combined. In practice, most projects start with naive or advanced RAG and grow to wherever the use case demands: which architecture carries the load is decided by your use case – not by the trend.
Quality you can measure
The difference between a demo chatbot and a productive knowledge system lies in quality assurance. For us, this includes three things:
- Evidence required: Every answer cites its sources. What cannot be backed by your documents is not claimed (groundedness).
- Measurable quality: Before the rollout, a set of test questions is created with the business departments. We use it to measure whether the search finds the right passages (retrieval precision) and whether the answers are factually correct.
- Continuous improvement: User feedback and unanswered questions flow systematically back into development – a knowledge system is never "finished", it keeps getting better.
Data protection, permissions and operation
Corporate knowledge is sensitive – and we treat it accordingly. Operation is GDPR-compliant and, if required, fully on-premises or in your own cloud environment; your documents then never leave your infrastructure. Equally important: the knowledge system respects your existing access rights. Anyone who is not allowed to read a document will not receive its content as an answer – permissions apply to the AI just as they do to every employee.
The path to your solution
Getting started is smaller than many think. This path has proven itself:
- Potential analysis: Together we identify the knowledge areas with the greatest benefit and clarify data situation, permissions and operating model.
- Pilot with real documents: A defined document set, real questions from the business departments, measurable results – in weeks, not months.
- Evaluation and expansion: What the pilot proves is expanded step by step – from naive RAG up to the stage your use case really needs.
The foundation for this is not so much the AI as your data – and that is exactly where our roots come into play: since 2005 we have been structuring corporate data, building data platforms and turning scattered repositories into reliable sources of information.