Every few weeks there's a new claim that AI can now write fully accessible interfaces on the first try. We didn't want to take a vendor's word for it, and we didn't want to read another press release. So we ran our own test, small enough to be honest about, and simple enough that anyone with the same four models could rerun it themselves.
We picked one UI pattern that quietly wrecks accessibility even for experienced developers: a custom-styled "Choose a country" combobox, meaning a button plus a styled option list, not a native <select>. We sent the exact same one-shot prompt to four different AI models available to us, took the raw HTML, CSS, and JavaScript each one produced, and graded it by hand against concrete WCAG success criteria. No follow-up prompts asking a model to fix its own mistakes. No cherry-picking the best run out of five attempts. Just the first answer, judged on its merits.
The Stat: A 2025 study introducing the FeedA11y method (arXiv 2503.15885) found that adding a feedback-driven review step meaningfully improves the accessibility of AI-generated code compared to a single, unreviewed prompt. (Source: arXiv 2503.15885)
The task we gave all four models
We asked for one thing: a self-contained HTML, CSS, and JavaScript "Choose a country" combobox, a trigger button that opens a styled list of options, built from scratch rather than a native <select>. This is a deliberately common request. Design systems ask for it constantly because a native select can't be restyled the way a brand wants, and it's exactly the kind of component where accessibility either gets built in from the start or gets bolted on badly later.
We gave each of the four models the identical prompt, one shot, no back-and-forth. Then we opened each result in a browser, unplugged the mouse, and worked through it with a keyboard and a screen reader.
What we actually graded
We didn't invent our own accessibility rubric. We checked each model's output against the W3C ARIA Authoring Practices Guide combobox pattern and three concrete success criteria:
- Keyboard operable end-to-end (WCAG 2.1.1): can you open, navigate, and select an option without ever touching a mouse?
- Correct programmatic name, role, and value (WCAG 4.1.2), including whether
aria-activedescendantactually points at the element that holds real DOM focus. This is the detail that trips up otherwise solid-looking code. - Visible focus indicator retained throughout (WCAG 2.4.7): does a sighted keyboard user always have a visual cue for where they are?
Model by model: what actually happened
Model A: the keyboard worked, but the screen reader never knew
Model A's combobox handled arrow keys, Enter, and Escape correctly, and because focus never left the trigger button, a sighted keyboard user could always see where they were. That's a real pass on keyboard operability and focus visibility.
But it had a genuine ARIA bug. It set aria-activedescendant on the option list container itself, an element that never receives actual DOM focus, since focus stayed on the trigger button the entire time. aria-activedescendant only works when it's placed on the element that currently holds focus. Put it anywhere else and a screen reader has no reliable way to know which option is supposedly "active." The keyboard worked. The announcement layer underneath it didn't.
Model B: the pattern that actually matches how screen readers work
Model B used role="combobox" on the trigger button and, critically, kept aria-activedescendant on that same button, the element that genuinely held focus the whole time. It also added Home and End key support, type-ahead (jump to an option by typing its first letter), and a live-region announcement when a selection was made. Of the four, this was the most complete implementation, and it lined up cleanly with the APG pattern linked above.
Model C: it looked identical until you unplugged the mouse
Model C's version looked fine in a demo. Click the button, options appear, click one, it's selected. But it shipped zero ARIA attributes and zero keyboard event handling of any kind. Tab to it, and nothing responds to arrow keys at all. There's no programmatic role announcing it as a combobox or listbox, so a screen reader user gets no indication it's even interactive beyond a generic clickable element. This is a textbook double failure: it fails keyboard operability (2.1.1) outright, and it fails name/role/value (4.1.2) because there's no role to expose in the first place. Since it never responds to a keyboard at all, there's no focus state worth evaluating either. We recorded that as a fail across the board, because a keyboard user is left with nothing to see or do.
Model D: real focus, correctly targeted
Model D took a different but equally valid approach. Instead of keeping focus on the trigger button, it moved actual DOM focus into the option list (tabindex="-1" plus a scripted .focus() call) before setting aria-activedescendant there. Because the listbox itself now genuinely holds focus, that placement of aria-activedescendant is correct. It also included full arrow key, Home/End, and type-ahead support. Along with Model B, this was a clean, internally consistent pass on all three criteria.
The scorecard, and what "2 out of 4" actually means
Two of the four models, B and D, shipped an internally consistent, keyboard-and-screen-reader-correct pattern on the first try, using two different but equally valid ways of handling focus and aria-activedescendant. One model, A, was fully keyboard-usable but had a real, screen-reader-breaking ARIA targeting bug that a purely visual or mouse-driven review would never catch. One model, C, skipped accessibility entirely and produced something that was functionally indistinguishable from the others if you only ever tested it with a mouse.
That last point is the one we'd underline. Every one of these four components would sail through a casual click-through demo. The differences only show up once you test with a keyboard and a screen reader, which is exactly the testing step most teams skip when they're moving fast with AI-generated code.
This is also exactly what the FeedA11y research points at: a single, unreviewed prompt is a coin flip, model by model, criterion by criterion. Ours was a one-shot test by design, so it's a fair illustration of that same baseline, not a follow-up-corrected best case. A feedback-driven review step, whether that's a second prompt pointing out the specific failure or a human running a real keyboard and screen reader pass, is what closes that gap.
What this means if you're shipping AI-generated components
If your team is generating UI code with AI, a few habits fall directly out of this test:
- Never trust a mouse-only demo. Model C proves a component can look completely finished and still be entirely unusable by keyboard.
- Check where
aria-activedescendantactually points, not just that it exists. Model A had it in the markup and it still didn't work, because it was on the wrong element. - Compare against the APG pattern for the component you're building, not just general accessibility advice, since combobox-specific pitfalls like this one are pattern-specific.
- Treat AI output as a first draft that needs a real review pass, not a finished component. That review step is the difference the FeedA11y research measured, and it's the difference we saw firsthand between Model B and Model D on one side and Model A and Model C on the other.
If you've followed our other coverage in this space, this test sits alongside a few related but distinct questions we've dug into. We looked at a single model building a single form conversationally in our ChatGPT accessible form test, which is a narrative deep dive rather than a structured, multi-model comparison. We separately tested accessibility overlay widgets, a completely different category of product, in AI overlays: we tested 5, all failed WCAG. And we covered what happens when you ask AI to repair an existing, already-built site rather than generate something new in I let ChatGPT fix my site's accessibility. This one is different from all three: it's a structured, criterion-by-criterion test of greenfield code generation across multiple models, graded by hand against the actual WCAG success criteria that matter.
If you want a fast, free way to see where your own site's components stand against the same kind of criteria we used here, our free accessibility tools are a good place to start, whether the code behind your site was written by a person, an AI model, or some mix of both.
None of this means AI can't write accessible code. Half of what we tested did, cleanly. It means the code needs the same scrutiny you'd give a junior developer's first pull request: a real keyboard pass, a real screen reader pass, and a check against the actual pattern spec, every single time.
Want a second set of eyes on it? Email experts@wcag.world and talk to a person on our team, or start with our free accessibility tools to see where you stand today.