How RAG connects a large language model (LLM) to your documents and systems, and what it takes to get secure, correct answers.
In short
RAG (retrieval augmented generation) lets an AI assistant answer from your company's own data. When a user asks a question, the system first searches your documents and systems for relevant passages, then gives them to a language model, which writes an answer based on those sources and can cite them. Quality depends mostly on your data and access rules.
Retrieval augmented generation is a way of building AI applications in which a language model looks up relevant information in a separate knowledge source before it answers, instead of relying only on what it learned during training. The term comes from a 2020 research paper by Lewis et al., and RAG has since become the standard pattern for company chatbots, internal search assistants and support tools built on generative AI (generativ AI, as Swedish teams often call it).
This guide is written by Devies, a Gothenburg-based product and engineering consultancy that builds data platforms and AI solutions for Nordic companies. It explains how the approach works, what your data needs to look like, how to handle security and access, how to measure answer quality and how to run a pilot that tells you something useful.
How does RAG work?
RAG works in two stages: a retrieval step that finds the most relevant passages in your own content, and a generation step in which an LLM writes an answer using only those passages as context.
In more detail, a typical setup has five parts:
- Ingestion. Documents, wiki pages, tickets and records are collected from source systems and split into smaller passages, often called chunks.
- Indexing. Each passage is converted into an embedding, a numeric representation of its meaning, and stored in a vector index together with metadata such as source, date and access group.
- Retrieval. The user's question is converted the same way, and the system fetches the passages that are closest in meaning. Many teams combine this with ordinary keyword search, which handles product codes and names better.
- Generation. The question and the retrieved passages are sent to the language model with instructions to answer only from the material provided.
- Citation. The answer shows which documents it is based on, so users can check it.
An LLM, or large language model, is a model trained on very large amounts of text to predict and generate language. On its own it knows nothing about your contracts, prices or internal processes, and it may produce fluent text that is wrong. Retrieval narrows the model's task from "know everything" to "summarise these five passages correctly", which is far easier to control.
Approach
What it does
Best for
Main limitation
Plain LLM chat
What it does
Answers from training data only
Best for
General writing and reasoning
Main limitation
No knowledge of your company, no sources
Retrieval augmented generation
What it does
Looks up your content, then answers
Best for
Policies, manuals, product data, support
Main limitation
Only as good as the content and access rules
Fine-tuning
What it does
Retrains the model on your examples
Best for
Tone, format, narrow repeated tasks
Main limitation
Costly to update, does not add reliable facts
Is your data ready?
Your data is ready for retrieval when the content that should answer the questions exists in writing, has a clear owner and current version, and can be reached by the system with its access rights intact.
Most failed pilots trace back to content, not models. Typical problems are three versions of the same policy, scanned PDFs without readable text, important knowledge that lives only in people's heads, and tables that lose their meaning when split into passages. Before you build, check the following:
- Completeness: do the documents that answer the top 20 questions actually exist?
- Currency: is there one valid version, and can old versions be excluded?
- Ownership: is someone responsible for keeping each source up to date?
- Structure: are headings, dates and document types consistent enough to be used as metadata?
- Structured data: will numbers come from a governed system such as an ERP or data platform, or from exported spreadsheets?
The last point matters. Questions such as "what was our margin on product X last quarter" should be answered from a reliable data source, not from a slide deck someone uploaded. That is a data governance question as much as an AI question, and we cover it in data governance for mid-sized companies.
Security and access control
A secure assistant must never show a user content that the same user could not open in the source system, so permissions have to follow every passage from source to answer.
The most common design mistake is to index everything into one shared collection and rely on the model to "not mention" sensitive content. That does not work. Filter retrieval by the user's identity and groups before anything reaches the model. Also consider:
- Prompt injection: a document or user input can contain hidden instructions that try to change the model's behaviour. As of October 2026, prompt injection is listed first (LLM01:2025) in the OWASP Top 10 for LLM Applications, which also lists vector and embedding weaknesses as LLM08:2025.
- Personal data: if passages contain personal data, the index is processing that data under the GDPR, including rules on purpose limitation and erasure. Deleting a record in the source must also remove it from the index.
- Data location and logging: know where the model provider processes prompts, whether they are stored, and what your own logs contain.
- Regulation: As of October 2026, the European Commission's AI Act page states that the AI Act became applicable on 2 August 2026, with AI literacy obligations applying since 2 February 2025. Most internal knowledge assistants are not high-risk systems, but you should document what the assistant does and train the people who use it.
How to evaluate answer quality
Answer quality is evaluated by testing retrieval and generation separately against a fixed set of real questions with known correct answers.
Build a test set of 50 to 100 questions collected from the people who will use the system, each with the expected answer and the document it should come from. Then measure:
- Retrieval hit rate: how often the correct passage is among the top results.
- Faithfulness: whether the answer is supported by the retrieved passages, without additions.
- Correctness: whether the answer matches what the expert says is right.
- Refusal behaviour: whether the assistant says "I don't know" when the content is missing, rather than guessing.
- Latency and cost per answer at expected volume.
The NIST Generative AI Profile (NIST AI 600-1) names confabulation, confidently stated false content, as one of the key risks of generative systems. A test set is how you detect it before users do. Rerun it every time you change the model, the chunking or the prompt, because small changes can shift results noticeably.
A pilot plan that tells you something
A useful pilot runs for six to ten weeks on one narrow use case, with a fixed test set, real users and a decision point at the end.
Weeks 1-2: choose and scope. Pick one domain with clear documents and a clear audience, for example product support, HR policies or internal technical documentation. Define the 50 to 100 test questions and what "good enough" means.
Weeks 3-5: build the minimum. Ingest one or two sources, apply access filtering from day one, and show citations in every answer.
Weeks 6-8: test with users. Let 10 to 20 people use it in daily work, log questions with no good answer, and fix content gaps rather than prompts.
Weeks 9-10: decide. Compare results with the targets you set and decide whether to scale, change scope or stop. Stopping is a valid outcome if the content is not there.
Keep reading
From our work
Client Case: StadsbyggnadsförvaltningenHow Devies helped Gothenburg's city planning office turn fragmented traffic data into a living, trusted Traffic Catalogue.Read the caseDevies
Want a second opinion?
Devies, a Gothenburg-based product and engineering consultancy, works on questions like this with Nordic product and IT teams. Tell us where you are and we will suggest a sensible first step.
Frequently asked questions
What is the difference between RAG and fine-tuning?
Retrieval augmented generation looks up relevant content from your documents each time a question is asked and gives it to the language model as context. Fine-tuning retrains the model on your examples so it changes its behaviour, tone or format. For answering questions from company knowledge that changes over time, retrieval is usually cheaper, easier to update and can show sources. Fine-tuning suits narrow, repeated tasks.
Can RAG stop an AI from hallucinating?
It reduces the problem but does not remove it. When the answer is based on retrieved passages, the model has less room to invent facts, and citations let users check the source. Errors still happen when retrieval finds the wrong passage, when content is outdated or when the model adds details. A fixed test set, clear refusal instructions and visible sources are the practical safeguards.
Is our data safe if we use an LLM with our internal documents?
It can be, if the solution is designed for it. Key measures are filtering retrieval by each user's existing permissions, choosing a model provider and region whose data processing terms you accept, controlling what is logged, and protecting against prompt injection. Personal data in the index is subject to the GDPR, so deletion and purpose rules must apply to the index as well as the source.
How long does a RAG pilot take?
A focused pilot on one use case typically takes six to ten weeks: two weeks to choose the scope and write test questions, about three weeks to build a minimal version with access control and citations, two to three weeks of testing with real users, and a final decision point. The timeline depends mostly on how ready the underlying documents and data are.
Sources
- Lewis et al., Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (2020), arXiv:2005.11401
- OWASP Gen AI Security Project, OWASP Top 10 for LLM Applications 2025
- NIST, Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile (NIST AI 600-1), July 2024
- European Commission, AI Act: regulatory framework and application timeline (checked 13 October 2026)
- Devies, Data and AI Use Case Discovery Workshop
- Devies, Data Hub
- Devies, Data Foundation Platform
