Skip to main content

BLACK FRIDAY Sale: 35% OFF Lumtera Pro — All Plan.

Offer ends in 0d 00h 00m

Get Lumtera Pro

Accuracy benchmark

How accurate are Lumtera's checks?

On the W3C ACT test cases, Lumtera reported 296 of 393 known failing examples at any severity. That count is generous: 121 were reported as errors, and the other 175 only ask a person to review or give a tip. Here is the method, the false-alarm test, and what the numbers do not show.

Run on November 5, 2026 with the code of Lumtera 1.3.0. Our own test, not an independent audit.

The numbers

  • 296 of 393 failing examples reported at any severity. A generous count: any word from a mapped check earns credit.
  • 121 of 393 failing examples reported as an error, the definite failures. The rest ask a person to decide.
  • 1 of 816 passing or inapplicable examples reported as an error by the check mapped to that rule.

Results on the W3C ACT test cases

Lumtera on the W3C ACT test cases
Measure Result
Failing examples reported at any severity 296 of 393
… counting only ACT rules that have a mapped Lumtera check 296 of 352
Failing examples reported as an error 121 of 393
Passing or inapplicable examples reported as an error by the mapped check 1 of 816
Passing or inapplicable examples given Needs review or a tip by the mapped check 107 of 816

Read 296 as generous. A failing example counts as reported when a mapped check says anything at all about that page. For 175 of them, Lumtera did not say "this fails": it asked a person to review, or gave a tip. A video check that asks someone to confirm captions earns credit on every ACT video rule, whether or not the captions are wrong, and the same page with good captions gets the same question. 121 is the number reported as definite failures. Neither figure is a share of all accessibility problems.

On the same passing and inapplicable examples, checks not mapped to that ACT rule reported 21 more errors from the new ARIA and keyboard checks. We read them: each is a real problem the example is not testing, such as a custom button nothing lets a keyboard reach.

We publish no comparison with other checkers in this run. Earlier versions of this page compared Lumtera with axe-core, using a mapping we never committed, so those figures could not be reproduced and we removed them. axe-core is a strong, well-established engine, and nothing here says Lumtera finds more than it does.

False alarms on accessible pages

We ran the checks on 532 pages that meet WCAG 2.2 AA or nearly so, then read every error. Review mode and the browser checks had 0 false alarms. The check of raw content had 8. An error on one of these pages counts as a false alarm unless it is a real problem in the page; Needs review does not count, because it asks a person to decide.

Errors reported on each set of accessible pages, by engine, before reading them
Pages Check of raw content Browser checks Review mode
Hand-written accessible pages (9) 0 0 0
WAI-ARIA Authoring Practices examples (76) 32 0 26
GOV.UK Design System component examples (248) 61 2 61
Twenty Twenty-Five: patterns and views (115) 0 0 0
Twenty Twenty-Four: patterns and views (75) 2 1 2
WooCommerce with Twenty Twenty-Five (9) Not run 0 1

These are all the errors reported, before reading them. Apart from the eight below, each is a real problem in the page as published: broken in-page links in the GOV.UK examples, untitled frames and missing link targets in the W3C pattern examples, two core output problems and one low-contrast core pattern in Twenty Twenty-Four, and a close button in WooCommerce's closed mini-cart that can still take focus.

The eight false alarms are in two W3C pattern examples whose script sets a button's name and hides an edit field while the page runs. The check of raw content reads the saved markup and cannot see that; review mode checks the page as the browser shows it and reports nothing there.

How we tested

  • The ACT corpus: every test case in the W3C ACT Rules repository, small example pages each marked as passing, failing or inapplicable. An ACT rule with no mapped Lumtera check counts as not found.
  • Each page went through the content checks, and through the whole-page checks in a real browser, including the keyboard walk and widget tests and a second run at 320 pixels wide for the checks that measure a width. Pages were served locally with other requests blocked.
  • Each ACT rule is mapped to the Lumtera checks that cover it, in a mapping file we keep with our tests. The file and the scripts are not public.
  • The accessible pages: WAI-ARIA Authoring Practices examples, GOV.UK Design System components, WordPress with Twenty Twenty-Five and Twenty Twenty-Four (every pattern included), WooCommerce pages and nine hand-written pages.
  • Review mode's column sends each page's browser snapshot through the same route review mode uses.

What these numbers do not show

  • Test cases are small, focused examples, not real sites.
  • 97 failing examples got no finding at all, 41 of them on ACT rules Lumtera has no check for. The benchmark lists the gaps.
  • Many WCAG criteria need a person: whether alt text is right, whether captions match the audio, whether reading and focus order make sense, whether error messages help. No automated checker can decide these.
  • Known limits remain: contrast inside shadow DOM is not measured yet, and the check of raw content cannot see changes a script makes after the page loads.
  • On the final 1.3.0 code we re-ran 524 of these pages, with one more Twenty Twenty-Five pattern than the table lists; the error counts did not change. The 9 WooCommerce pages come from the run on 4 November 2026.
  • Not re-run for this release: Lumtera Pro's rendered measures over these pages, and the content check on the WooCommerce pages. Screen readers and real user testing are not part of this benchmark.

How we ran it

Our test scripts and the rule mapping are not public, so nobody outside MantraBrain can repeat this exact run. The W3C ACT test cases are public, so you can try any of them against Lumtera yourself.

  1. We fetch the third-party pages from their own sources. They keep their own licences.
  2. On a test site with Lumtera active, we run the content checks over the pages, and the browser checks in headless Chrome, once at normal width with the on-demand tests and once at 320 pixels.
  3. We score ACT using the rule-to-check mapping.
  4. For review mode's column, we send the browser snapshots through review mode's own route as an administrator.

Sources

See what it finds on your site

The free plugin has 88 content checks and 38 whole-page checks; Pro adds its own. A finding that is not certain is marked Needs review, with one exception: an Image block with empty alt text and no sign of being decorative is an error, because that is often a field left empty. Mark the image decorative and it clears.

Start with the free plugin today

All 88 content checks and 38 whole-page checks, fixes you approve with undo, and the site report — free, with no account and no page limits.