You ran the scanner. It came back clean. Zero errors, green checkmarks across the board.
You told your team the site passes. You maybe even told a customer, or wrote it into a VPAT. Here's the uncomfortable truth: that clean report proves almost nothing, because automated tools are structurally incapable of evaluating most of what WCAG actually requires.
Picture This
A product manager at a mid-size SaaS company gets a support ticket: a screen reader user can't complete checkout. She's confused — the team ran axe DevTools on every page last quarter, and it came back essentially clean. Maybe a few contrast warnings, all fixed.
So she pulls up the checkout flow herself with a screen reader running for the first time. Thirty seconds in, she understands. The "Continue" button has an accessible name, technically. It says "Continue." The scanner saw a button with a name and moved on. What the scanner couldn't know is that there are four "Continue" buttons on adjacent steps, none distinguishable from each other out of context, and the focus order jumps from the shipping form straight to a promo code field, skipping the payment section entirely.
Nothing here throws an error. Nothing here is invalid HTML or a missing attribute. It's just broken — in a way only a human walking through the flow would ever notice.
Why Scanners Miss This by Design, Not by Bug
This isn't a limitation the tool makers hide. It's stated plainly in their own documentation: automated testing can only reliably evaluate a subset of WCAG success criteria, because a large portion of the guidelines require human judgment a script cannot make.
Tools like axe, WAVE, and Lighthouse are genuinely useful. They're fast, consistent, and excellent at catching a real category of issues: missing alt attributes, insufficient color contrast on static text, missing form labels, invalid ARIA attribute values. If your scan flags these, fix them — they're real problems.
But look at what a scanner is actually checking versus what the criterion actually requires.
Take 1.1.1 Non-text Content. A scanner can confirm an <img> tag has an alt attribute. It cannot tell you whether that attribute says "Founder photo, smiling, arms crossed" or "IMG_4821.jpg." Both pass the automated check. Only one is useful to a blind user.
Take 2.4.3 Focus Order. A scanner can confirm every interactive element is technically reachable by keyboard, in some sequence. It has no way to judge whether that sequence is logical to a human tabbing through a form — whether it jumps around the page, skips a field, or dumps focus somewhere unexpected after a modal closes.
And no scanner can tell you whether a screen reader user can actually complete checkout, sign up for an account, or file a support ticket. That requires someone using assistive technology the way a real user would, start to finish, and noticing where it breaks down.
The Stat: WebAIM's annual "WebAIM Million" evaluation of the top 1,000,000 home pages has repeatedly found the vast majority — in recent years around 95-96% — have detectable WCAG 2 failures, with low-contrast text and missing alt text consistently among the most common. If failures this basic and this automatable still show up at that rate, the harder, judgment-based failures automation can't even see are almost certainly still there too. (WebAIM: The WebAIM Million)
What Manual Testing Actually Fills In
Manual testing isn't vague "have a human look at it" work. It's a specific methodology built to cover exactly the gaps automation leaves open.
Real keyboard-only navigation through your key user flows — not a single page, but the paths that matter: sign-up, checkout, account settings, whatever your users actually do. Does Tab order follow a sensible sequence? Does focus ever get trapped inside a modal, or vanish entirely after an action completes?
Real screen reader testing — NVDA, JAWS, VoiceOver (we cover the specifics of running these in our screen reader testing article) — walking through the same flows a keyboard tester covers, confirming that content and controls are not just present in the DOM, but actually understandable and operable in the order and language a screen reader announces them.
Neither of these is something a script can simulate. They require a person experiencing the interface the way an assistive technology user experiences it, and making a judgment call about whether it works.
The Programs That Actually Work Layer Both
This isn't automated testing versus manual testing. Treating either one alone as sufficient leaves a real, documented gap — the effective programs combine them deliberately.
Automated scanning belongs in CI/CD, running on every deploy. It's fast, it's cheap, and it's excellent at catching regressions — someone removes an alt attribute in a refactor, ships a new component with insufficient contrast, or breaks a form label, and the scan catches it before it reaches production.
Manual and assistive-technology audits belong on a periodic cycle, focused on your highest-traffic and highest-stakes flows. These are slower and more expensive per hour, but they're the only method that catches focus order problems, meaningless labels, ambiguous alt text, and broken end-to-end task completion — the failures that automation is structurally unable to see.
Comparison: What Each Method Actually Catches
| Failure type | Caught by automated scan | Caught by manual/AT testing |
|---|---|---|
Missing alt attribute |
Yes | Yes |
| Alt text present but meaningless ("IMG_4821.jpg") | No | Yes |
| Insufficient color contrast on static text | Yes | Yes |
| Missing form label (markup-level) | Yes | Yes |
| Illogical keyboard focus order | No | Yes |
| Focus trapped or lost after modal closes | Partial | Yes |
| Invalid ARIA attribute value | Yes | Yes |
| ARIA present but confusing when announced by a screen reader | No | Yes |
| Screen reader user can complete a multi-step task (e.g. checkout) | No | Yes |
| Regression introduced in a new deploy | Yes | No (too slow for this) |
The Compliance Claim You Can't Actually Make
Here's where this stops being a QA nuance and starts being a liability question: a completely clean automated scan report is not the same claim as "this site is WCAG conformant." Presenting it that way — to a customer, to legal, in a VPAT — creates exactly the kind of overconfident, easily-disproven claim we cover in our accessibility statement article. A plaintiff's attorney, or a customer's own accessibility team, doesn't need much more than a screen reader and ten minutes on your checkout page to disprove it.
And the pressure on this front isn't going away. UsableNet's annual ADA Digital Accessibility Lawsuit Report has tracked several thousand federal ADA website lawsuits filed per year in the U.S. in recent years, with plaintiffs' firms increasingly sending pre-suit demand letters as well as filing suit — and retail/ecommerce is consistently reported among the most-sued industries in those same reports. A clean scanner report will not be the thing that protects you if the claim is "a screen reader user could not complete checkout."
What to Actually Do About It
You don't need to abandon your scanning tools. Use them for exactly what they're good at: continuous regression catching in CI/CD, and fast triage of the obvious, high-volume issues — missing alt text, contrast, labels, invalid ARIA. That work is real and worth automating.
Then budget separately for genuine manual and assistive-technology testing of your highest-traffic and highest-stakes flows. Not your whole site, necessarily — your checkout, your signup, your core product task. That's where a real person using a keyboard and a screen reader will find the failures your scanner structurally cannot.
A clean scan is a starting point. It was never the finish line.
Get the Testing Your Scanner Can't Do
If the last accessibility "audit" your site got was a green checkmark from a browser extension, you don't actually know where you stand. Our team runs real keyboard-only and screen reader testing on your highest-stakes flows — checkout, signup, the paths your business depends on — layered on top of whatever automated scanning you already have in place.
Get a real manual audit layered on top of your automated scans.