What makes language-model applications different to secure, how prompt injection and data leakage happen, and the design decisions that keep the damage small.
In short
Generative AI security is the work of stopping a language-model application from being manipulated, leaking data or taking harmful actions. The two main risks are prompt injection, where text in a prompt or document overrides your instructions, and data leakage. Neither can be filtered away completely, so limit what the model can see and do.
Generative AI security, often searched in Sweden simply as AI security (AI-säkerhet), is not a new discipline so much as an old one with a new weak spot. Devies, a Gothenburg-based product and engineering consultancy, builds assistants, retrieval systems and agents for companies that want them in production, and the same question comes up in every project: what happens when someone tries to make the model misbehave? This explainer is for CTOs, security leads and product owners who need to understand the risks well enough to make design decisions, not only to buy a filter.
What is generative AI security?
Generative AI security is the set of controls that protect an application built on a large language model (LLM) from manipulation, data exposure and unintended actions. It covers the model, the prompts, the data the model can reach, the tools it can call and what happens to its output.
The difference from ordinary application security is structural. A language model receives instructions and data in the same channel: plain text. Classic injection flaws were fixed by separating the two, as parameterised queries do for SQL. The UK National Cyber Security Centre argues that this separation does not exist inside an LLM, and that prompt injection is not SQL injection and may be worse because it may never be fully mitigated. The practical conclusion is to design for containment, not for perfect prevention.
The OWASP Top 10 for LLM Applications (2025) is the most widely used risk list. It ranks prompt injection first and sensitive information disclosure second, followed by supply chain, data and model poisoning, improper output handling, excessive agency, system prompt leakage, vector and embedding weaknesses, misinformation and unbounded consumption. For the web-application equivalent, see our OWASP Top 10 explainer for product and engineering leads.
How does prompt injection work?
Prompt injection is an attack where text supplied to the model changes its behaviour against the developer's intent. It comes in two forms, and the second is the one most teams underestimate.
- Direct prompt injection is typed by the user: "ignore your previous instructions", role-play, fake conversation turns or encoded text that slips past rules. Microsoft calls these user prompt attacks.
- Indirect prompt injection hides instructions in content the model reads on the user's behalf: a web page, an email, a PDF, a support ticket or a document in your knowledge base. Microsoft calls these document attacks, and its Prompt Shields documentation lists goals such as data exfiltration, fraud, malware links and blocking the system's capabilities.
Indirect injection matters because the attacker never needs access to your application. If your assistant summarises incoming email, anyone who can send you an email can try to give it orders. The risk grows with every tool the model can call: an assistant that can only answer is embarrassing when hijacked; one that can send messages, update records or run code is dangerous.
The following table separates prompt injection from things it is often confused with.
Term
What it is
How it differs from prompt injection
Jailbreak
What it is
A direct injection aimed at bypassing the model's safety training
How it differs from prompt injection
A subset: always typed by the user, usually aimed at content rules
Hallucination
What it is
The model produces confident but wrong output
How it differs from prompt injection
No attacker involved; a quality problem, not manipulation
Data poisoning
What it is
Tampering with training or fine-tuning data
How it differs from prompt injection
Changes the model itself before use, not the input at run time
System prompt leakage
What it is
The model reveals its hidden instructions
How it differs from prompt injection
Often a result of injection; harmful only if secrets sit in the prompt
SQL injection
What it is
Malicious input interpreted as a database command
How it differs from prompt injection
Fixed by separating data from code, which LLMs cannot do
How does data leak from an AI application?
Data leakage in a generative AI system is any path by which information reaches a person or system that should not have it. Most leaks are not exotic attacks but ordinary access-control mistakes made more visible by a fluent interface.
The common paths are:
- Over-broad retrieval. A retrieval-augmented assistant indexes documents with a service account that sees everything, then answers anyone. Permissions must be enforced at retrieval time, per user, as described in our guide to RAG on your own company data.
- Secrets in the prompt. API keys, internal URLs or business rules in the system prompt will eventually be extracted. Treat the system prompt as public.
- Exfiltration through output. An injected instruction asks the model to embed data in a link or image URL that the user's browser then loads, sending the data to the attacker.
- Logs and telemetry. Prompts and answers are stored for debugging and evaluation, often with weaker controls than the source systems.
- Third-party processing. Data sent to a model provider is subject to that provider's retention and location terms, which belong in your data processing agreement.
Which controls actually reduce the risk?
The controls that work best limit what a compromised model can reach and do, because no input filter catches every attack. Think in layers, from architecture down to monitoring.
Design controls
- Least privilege for the model. Give it access only to the data and tools the current user is allowed to use, through the user's own identity rather than a shared super-account.
- Narrow tools instead of open access. Expose specific, parameterised functions instead of letting the model write free-form queries. Devies used this approach when building the SEVR Insights accounting platform: a custom query language gives the AI a controlled set of functions that return pre-aggregated, deterministic results scoped to each client, so the model cannot browse financial data freely.
- Human approval for consequential actions. Sending, paying, deleting and publishing should need a confirmation that shows exactly what will happen.
- Treat output as untrusted input. Encode it before rendering, never execute it directly, and block automatic loading of external links and images.
Detection and operational controls
- Input screening such as Prompt Shields for user prompts and documents. Microsoft notes that document attack detection is off by default and that the models are primarily trained on English content, so test Swedish inputs separately.
- Rate and cost limits against unbounded consumption.
- Logging with access control and retention rules, plus alerts for unusual tool calls.
- Red-team testing before launch with a written set of injection cases, rerun at every model or prompt change.
What should you do first?
Start with an inventory and a threat model for each AI application, then fix the design issues that give a hijacked model the most power. A practical first pass looks like this:
- List every generative AI application, including internal pilots and embedded vendor features.
- For each one, record which data it can read, which tools it can call and whose identity it uses.
- Map untrusted inputs: user text, uploaded files, retrieved documents, email and web content.
- Remove secrets from prompts and move credentials to a secrets store.
- Replace broad data access with per-user permissions and narrow functions.
- Add confirmation steps for any action with external effect.
- Write injection test cases and run them before each release.
- Assign an owner who reviews logs and incidents.
If the application acts autonomously, apply the same thinking to its whole workflow; our guide to AI agents from pilot to production covers guardrails and accountability in more detail.
Regulation points the same way. As of October 2026, the EU AI Act requires high-risk systems to be robust and secure, with the Annex III obligations applying from 2 December 2027 after the AI Omnibus amendment. Most business assistants are not high-risk, but robustness against manipulation is becoming a baseline expectation from customers and auditors anyway.
Keep reading
From our work
Client Case: HedinFaster loads and clean code: bringing modern software habits to Hedin’s BI platform.Read the caseDevies
Want a second opinion?
Devies, a Gothenburg-based product and engineering consultancy, works on questions like this with Nordic product and IT teams. Tell us where you are and we will suggest a sensible first step.
Frequently asked questions
Can prompt injection be completely prevented?
No. Because a language model reads instructions and data as the same kind of text, there is no reliable way to stop every injected instruction. The UK NCSC argues it may never be fully mitigated in the way SQL injection was. The realistic goal is to reduce the impact: limit the data and tools the model can reach, require human approval for consequential actions and treat its output as untrusted.
What is the difference between direct and indirect prompt injection?
Direct prompt injection is typed by the user into the chat, for example asking the model to ignore its rules or adopt another persona. Indirect prompt injection is hidden in content the model reads, such as an email, web page or document in a knowledge base. Indirect attacks are more dangerous for business systems because the attacker never needs access to your application, only a way to get text in front of it.
Is our company data safe if we use a public AI chatbot?
It depends on the service terms and how you use it. Business offerings from major providers usually state that customer prompts are not used to train their models and offer data processing agreements, while consumer versions may differ. The larger risk is often internal: staff pasting sensitive data into unapproved tools. Set a policy, provide an approved tool and check retention and data location terms before rollout.
Do input filters like Prompt Shields solve the problem?
They help but do not solve it. Classifiers such as Microsoft Prompt Shields detect many known attack patterns in user prompts and documents, and they are worth enabling. Microsoft itself notes they do not judge whether content is harmful, accurate or appropriate for the task, and performance can vary for non-English content. Use them as one layer alongside least privilege, narrow tools, approval steps and testing.
Sources
- OWASP GenAI Security Project: OWASP Top 10 for LLM Applications 2025
- UK NCSC: Prompt injection is not SQL injection (it may be worse), 8 December 2025
- Microsoft Learn: Prompt Shields in Azure AI Content Safety
- European Commission: AI Act, regulatory framework and application timeline
- Devies: SEVR Insights client case
