The difference between a penetration test, vulnerability scanning, threat modeling and red teaming, and when each method pays off.
In short
Threat modeling finds design flaws before code exists. Vulnerability scanning finds known weaknesses automatically and continuously. A pentest has skilled people try to exploit a finished system. A red team tests whether your organisation detects and stops a realistic attack. Most product companies need the first three, in that order.
The four methods answer different questions, so choosing between them is less about which is best and more about where you are in the product lifecycle and what you need to prove. Buying a penetration test for a system that has never been scanned wastes expert hours on findings a tool would have caught. Running only scanners misses the logic flaws that only a person or a design review will find.
This guide is written by Devies, a Gothenburg-based product and engineering consultancy that works with product cybersecurity and secure design. It compares the methods side by side, places them in the lifecycle, explains what drives cost and shows how to combine them.
How do the four methods compare?
The four methods differ in what they look at, who does the work and when in the lifecycle they give the most value.
First, the definitions:
- Threat modeling is a structured review of a system's design to identify what can go wrong, which threats matter and which controls are needed. The OWASP Threat Modeling guidance describes it as an activity that can start before any code is written.
- Vulnerability scanning (sårbarhetsskanning) is the automated checking of code, dependencies, containers or hosts against databases of known weaknesses.
- Vulnerability analysis (sårbarhetsanalys) is the broader work of assessing scan results and other findings: which are real, exploitable and worth fixing first.
- A penetration test is a time-boxed attempt by skilled testers to exploit a system the way an attacker would, within an agreed scope. NIST SP 800-115 describes it as security testing that mimics real-world attacks.
- A red team exercise is a goal-based simulated attack against the whole organisation, including people and processes, to test detection and response rather than to list every flaw.
Threat modeling
Vulnerability scanning
Penetration test
Red team
Question answered
Threat modeling
What could go wrong in this design?
Vulnerability scanning
Do we have known weaknesses?
Penetration test
Can someone exploit this system?
Red team
Would we detect and stop a real attack?
Done by
Threat modeling
Your team, facilitated
Vulnerability scanning
Tools, with human triage
Penetration test
External or internal specialists
Red team
Specialised external team
Best timing
Threat modeling
Design and major changes
Vulnerability scanning
Continuously, every build
Penetration test
Before release, after big changes
Red team
Mature organisations, periodically
Typical output
Threat modeling
Threats, risks, required controls
Vulnerability scanning
Ranked list of known issues
Penetration test
Exploited findings with evidence
Red team
Attack narrative, detection gaps
Finds design flaws
Threat modeling
Yes
Vulnerability scanning
No
Penetration test
Sometimes
Red team
Sometimes
Effort
Threat modeling
Hours to days
Vulnerability scanning
Low once automated
Penetration test
Days to weeks
Red team
Weeks to months
When should you run a pentest, and when something else?
Run a pentest when the system is built and stable enough to attack; use threat modeling earlier and vulnerability scanning all the time.
A practical mapping to the product lifecycle:
- Concept and architecture. Threat modeling gives the highest return here, because changing a design on a whiteboard costs a fraction of changing it in production. Revisit the model when you add an interface, a data flow or a third-party integration.
- Development. Vulnerability scanning belongs in the CI pipeline: dependency checks, static analysis, container and secrets scanning. Combined with a software bill of materials, it shows which components are affected when a new advisory appears. Our guide on what an SBOM is and why the CRA requires it covers that inventory.
- Before release. A penetration test validates that the controls you designed actually hold, and finds chained or logic flaws that scanners cannot see. Schedule it once the scope is stable, not during the last week before launch.
- In operation. Keep scanning, retest after significant changes, and consider periodic penetration tests for exposed systems.
- Mature security organisation. A red team exercise makes sense once you have monitoring and incident response to test. Without them, the result is predictable and expensive.
For EU product companies there is also a regulatory reason to plan this. The Cyber Resilience Act, Regulation (EU) 2024/2847, requires manufacturers to apply effective and regular tests and reviews of the security of their products. As of October 2026, its vulnerability reporting obligations have applied since 11 September 2026 and the main requirements apply from 11 December 2027, according to EUR-Lex. The act does not prescribe one test method, so the combination above is a reasonable way to show due diligence.
Financial entities have a more specific rule. The Digital Operational Resilience Act (DORA) requires some of them to run threat-led penetration testing, and the ECB's TIBER-EU framework was updated in 2024 to align with those requirements.
What drives the cost of each method?
Cost is driven mainly by scope, depth and how much expert time the method needs, not by the name of the service.
- Threat modeling costs mostly your own team's time plus a facilitator. A focused session on one system or feature is short; modelling a whole platform takes longer. The cheapest flaw to fix is the one found here.
- Vulnerability scanning has a tool cost and a setup cost, then a running cost in triage. The hidden expense is noise: without vulnerability analysis to separate real risks from false positives, teams stop reading the reports.
- A penetration test is priced on scope (number of applications, APIs, user roles and environments), test type (black box, grey box or white box) and retesting. Clear scope and access shorten the work considerably.
- A red team engagement is the most expensive, because it runs over weeks, combines several techniques and needs coordination with your security operations.
Two things reduce the bill for all four. Fix the obvious issues that scanners find before the expensive tests start, and give testers the threat model so they spend time on the risks that matter.
How do you combine the methods?
The methods work best as a chain where each one feeds the next: the design review sets priorities, scanning removes known issues and the penetration test checks what is left.
A checklist for a product team:
- Model the threats for each new product or major feature, and record the controls you decide on.
- Automate scanning of code, dependencies and containers in CI, and set clear rules on which severities block a release.
- Triage continuously. Assign owners and deadlines, and document why a finding is or is not exploitable.
- Scope penetration tests from the threat model, so testers target the high-risk flows such as authentication, payments or device control.
- Feed results back. Every penetration test finding should update the threat model and, where possible, become an automated check.
- Add a red team exercise only when detection and response are in place and the earlier steps run routinely.
Keep reading
From our work
Consilium and Devies Cybersecurity+Consilium Safety Group and Devies Cybersecurity+: a partnership for future-proof cybersecurity in fire, flame and gas safety systems.Read the caseDevies
Want a second opinion?
Devies, a Gothenburg-based product and engineering consultancy, works on questions like this with Nordic product and IT teams. Tell us where you are and we will suggest a sensible first step.
Frequently asked questions
What is the difference between a pentest and a vulnerability scan?
A vulnerability scan is automated: a tool compares your code, dependencies or hosts against databases of known weaknesses and produces a list. A pentest is manual: skilled testers try to exploit the system, chain weaknesses together and find logic flaws no tool knows about. Scans are cheap and should run continuously. Penetration tests are deeper, cost more and are typically run before release and after major changes.
Is threat modeling better than a penetration test?
Neither replaces the other. Threat modeling reviews the design early and finds flaws before they are built, when fixes are cheap. A penetration test checks the finished system and proves whether attacks actually work. The strongest approach uses the threat model to decide what the penetration test should focus on, then feeds the test findings back into the model for the next release.
How often should we run a penetration test?
A common practice is at least once a year for internet-facing systems, plus after significant changes such as a new authentication flow, a new API or a major architecture change. The right frequency depends on risk, regulation and how often you release. Teams that ship continuously usually rely on automated scanning between tests and retest only the parts that changed.
When does a red team exercise make sense?
A red team exercise makes sense when you already have security monitoring, incident response and routine vulnerability management in place. It tests whether people, processes and tools detect and stop a realistic, goal-based attack. For an organisation without those capabilities, a red team mostly confirms known gaps at a high price, so start with threat modeling, scanning and penetration tests.
Sources
- OWASP, Threat Modeling
- NIST SP 800-115, Technical Guide to Information Security Testing and Assessment
- Regulation (EU) 2024/2847 (Cyber Resilience Act), EUR-Lex
- European Central Bank, TIBER-EU
- Devies Cybersecurity+
- Devies, The Cyber Resilience Act: Alignment Workshop
