What an automated checker can and cannot tell you, what a manual audit adds, and how to decide which one your product needs now.
In short
Automated accessibility testing is fast, cheap and repeatable, but it finds only part of the barriers and cannot confirm conformance. A manual audit uses keyboard, screen reader and expert judgement against every WCAG criterion in scope. Most teams need both: a scanner in the pipeline for regressions, and a manual audit before launch, redesign or a legal deadline.
Accessibility testing is the work of checking whether people with disabilities can perceive, operate and understand a digital product, usually measured against the Web Content Accessibility Guidelines (WCAG). This comparison is written by Devies, a Gothenburg-based product and engineering consultancy, for product owners and engineering leads who need to decide where to spend a limited budget: on tools, on an expert review, or on a mix of the two.
What is the difference between automated and manual accessibility testing?
Automated testing runs code rules against the rendered page, while a manual audit is a person evaluating the interface with real input methods and assistive technology against each success criterion.
An accessibility checker is a tool that inspects the HTML, styles and accessibility tree of a page and flags patterns that break a rule, such as an image without a text alternative or text below the required contrast ratio. It returns the same result every time for the same code, which is exactly why it is useful in a build pipeline.
An accessibility audit is a structured evaluation by a trained person who works through a defined scope, uses keyboard, screen reader and zoom, and judges each WCAG criterion as passed, failed or not applicable. A WCAG audit is the same thing with the standard and level stated explicitly, for example WCAG 2.2 level AA.
W3C is clear about the limits of tools: they cannot check all accessibility aspects automatically, and human judgement is required. Tools can only assist an evaluation; they cannot determine whether a site is accessible.
How much does an automated accessibility checker actually find?
An automated checker finds a meaningful share of barriers, but on its own it misses a large minority of them and cannot tell you whether you conform.
The UK Government Digital Service tested this directly. It built a page with 143 deliberate barriers and ran common tools against it. In its published results, the best single tool found 37 percent counting errors and warnings, or 41 percent when its prompts for manual inspection were included. All tools together found about 71 percent. The study is from 2017 and tools have improved since, but the shape of the result still holds: no single tool covers everything.
What tools do well:
- missing text alternatives, empty links and empty buttons;
- colour contrast on static text;
- form fields without a programmatic label;
- missing document language, duplicate IDs and invalid ARIA attributes.
What needs a person:
- whether alternative text actually describes the image, or a link text makes sense out of context;
- a logical focus order and a visible focus indicator in every state;
- whether a screen reader announces names, roles, states and error messages in a usable way;
- WCAG 2.2 checks such as drag-only interactions, focus hidden behind sticky elements and login that blocks paste;
- whether a whole journey, like checkout or registration, can be completed.
A clean scan therefore means "no detectable rule failures", not "accessible".
Automated test or manual audit: decision table
The right choice depends on the question you need answered, how often you need the answer, and what you must be able to show afterwards.
Criterion
Automated testing
Manual audit
Question it answers
Automated testing
Did this code break a known rule?
Manual audit
Can people with disabilities use this product, and which criteria fail?
Coverage of WCAG
Automated testing
Partial, rule-based
Manual audit
Every criterion in scope
Speed
Automated testing
Seconds to minutes per page
Manual audit
Days, depending on scope
Cost per run
Automated testing
Low once set up
Manual audit
Higher, paid per scope
Repeatability
Automated testing
Identical every run
Manual audit
Depends on method and evaluator
Finds journey-level problems
Automated testing
Rarely
Manual audit
Yes
Supports a conformance claim or accessibility statement
Automated testing
No, not on its own
Manual audit
Yes, with documented scope and method
Best place in the process
Automated testing
Pull requests, CI and monitoring
Manual audit
Before launch, after redesign, before a legal deadline
Choose automated testing when
- you want to stop regressions on every pull request or nightly build;
- you need a quick first picture of many pages before deciding where to dig;
- you are fixing a backlog and want fast confirmation that known issues are gone.
Choose a manual audit when
- you must publish an accessibility statement or compliance information and need evidence behind it;
- you are launching, redesigning or replacing a design system;
- a customer, procurement process or regulator asks for a WCAG test report;
- users report barriers that your scanner does not show.
How do you combine both in practice?
The most cost-effective setup uses tools for breadth and continuity and people for depth at the moments that matter.
- Screen first. Run an automated scan across your key templates to remove the obvious failures before anyone spends audit hours on them.
- Define the audit scope. Name the WCAG version and level, the browsers and devices, and the critical journeys. W3C's evaluation methodology, WCAG-EM, describes how to pick a representative sample.
- Audit manually. Test each journey with keyboard only, a screen reader on desktop and mobile, 200 percent zoom and a 320-pixel viewport.
- Turn findings into backlog items. Each finding needs the location, steps to reproduce, the criterion and the user impact, so developers can fix it without a second meeting.
- Automate what you learned. Add rules and component tests for the patterns that failed, so the same problem cannot come back silently.
- Re-audit on change. Repeat the manual part for affected journeys at major releases, not on a fixed calendar alone.
The point of step 1 is money: an auditor who spends the first day listing missing alt text is doing work a scanner does in seconds. The point of step 5 is the reverse: every manual finding that can be expressed as a rule should become one.
Which accessibility testing do you need right now?
Your current situation decides the first step, and for most organisations that is a scan followed by a scoped audit of the journeys that carry the business.
- No testing yet: start with a free scan to size the problem, then audit your two or three most important journeys.
- Tools in place, no audit: your scanner is probably passing while real barriers remain. Commission a manual audit of the critical journeys.
- Audit done a year or more ago: check what has changed since. New components, a new checkout or a new login flow each justify a fresh manual round.
- Legal deadline ahead: prioritise a WCAG audit with documented scope and method, because that is what an accessibility statement or compliance file has to rest on.
As of October 2026, Swedish and EU rules refer to EN 301 549, which is currently cited in a version based on WCAG 2.1 AA, while the newer version based on 2.2 AA awaits citation. Our guide to WCAG 2.2 and how to test it explains what that means for your test target.
Keep reading
From our work
Client Case: Lea BankLea Bank rebuilt its digital platform with Strapi, React, GrowthBook, GA4, and BigQuery to improve SEO, A/B testing, and marketing autonomy.Read the caseDevies
Want a second opinion?
Devies, a Gothenburg-based product and engineering consultancy, works on questions like this with Nordic product and IT teams. Tell us where you are and we will suggest a sensible first step.
Frequently asked questions
Can an accessibility checker make my website WCAG compliant?
No. An accessibility checker finds rule-based failures such as missing alternative text, low contrast and unlabelled form fields, but many WCAG criteria require human judgement, including focus order, meaningful link text and screen-reader behaviour. A site can pass every automated rule and still be unusable with a keyboard. Use the checker to catch regressions, and a manual audit to establish conformance.
What does a WCAG audit include?
A WCAG audit states the standard and level, for example WCAG 2.2 AA, and a defined scope of pages, components and journeys. The evaluator tests each in-scope success criterion with keyboard, screen reader, zoom and code inspection, then reports each failure with its location, the criterion, the user impact and how to reproduce it. A good audit ends with a retest of fixed items.
How often should we run accessibility testing?
Run automated checks continuously, on every pull request or at least nightly, because they are cheap and catch regressions early. Run a manual audit at launch, after a redesign and when critical journeys such as login, checkout or registration change. Many organisations also review their accessibility statement or compliance information once a year, which is a natural point to retest key journeys.
Is a free WCAG test enough for a public-sector website?
A free WCAG test is a good starting point but not enough on its own for a Swedish public-sector website. The law on digital public services requires an accessibility statement that describes how well the service meets the requirements, and that statement needs evidence from a review covering criteria a tool cannot judge. Combine an automated scan with a manual review of your key services.
Sources
- W3C WAI: Selecting Web Accessibility Evaluation Tools
- W3C: Website Accessibility Conformance Evaluation Methodology (WCAG-EM)
- UK Government Digital Service, Accessibility blog: What we found when we tested tools on the world's least-accessible webpage (24 February 2017)
- W3C: Web Content Accessibility Guidelines (WCAG) 2.2
- DIGG: Digital offentlig service ska vara tillgänglig
