Manual audit vs automated testing
What a manual accessibility audit catches that a scanner cannot
A clean automated scan is not the same as WCAG conformance. Automated testing is very good at machine-checkable failures and structurally unable to judge quality or intent. This page sets out where the line falls, so you can tell which one your situation actually needs.
Scope note. The reviews collected on this site are about WebAbility.io's self-serve platform — the scanner, the automatic fixes and the widget. None of them evaluate an expert manual audit engagement, so nothing on this page claims review evidence for one.
The same criterion, two different jobs
| WCAG area | What an automated scan does | What a human has to do |
|---|---|---|
| Images have alt text | Detects a missing alt attribute | Judges whether the alt text describes the image, and whether a decorative image is correctly hidden |
| Colour contrast | Measures the ratio and flags failures | Catches text over gradients, video and background images, and hover or focus states the scanner never rendered |
| Form fields are labelled | Detects an input with no accessible name | Judges whether the label is understandable, and whether the error message says how to fix the problem |
| Focus order | Detects positive tabindex values | Follows the actual tab path and decides whether the sequence is logical and nothing is trapped |
| Headings | Detects skipped levels and empty headings | Judges whether headings describe the content beneath them |
| Link purpose | Detects an empty link | Decides whether the link text makes sense out of context — the reason a page of “read more” links fails |
| Custom widgets | Validates ARIA roles and attributes | Operates the component with a keyboard and a screen reader to see whether it is genuinely usable |
The pattern is consistent: the scanner answers does this attribute exist, a person answers does it do its job. Both matter. Neither replaces the other.
When you need the human pass
- A legal deadline or a demand letter. A scan report is not a conformance claim and should never be presented as one.
- Procurement. Buyers — particularly public sector and education — ask for an Accessibility Conformance Report or VPAT, which requires per-criterion findings a human signed off.
- Custom interactive components. A bespoke date picker, a multi-step form, a modal flow, a data grid. Automated tools validate the ARIA and still miss whether the thing is operable.
- Content-heavy sites. Alt text quality, heading sense and link text are judgement calls at every instance.
What the deliverable should contain
Any audit worth paying for should give you, at minimum:
- The standard and level tested against, stated explicitly (for example WCAG 2.2 Level AA)
- The pages, flows, browsers and assistive technology in scope
- Each finding mapped to the specific success criterion it breaches
- Severity, so the list is a work queue and not an undifferentiated dump
- Remediation guidance per finding, specific enough for a developer to act on
- Enough reproduction detail to verify the fix afterwards
If a report gives you a score and a issue count with no criterion mapping and no remediation steps, it is a scan with a cover page.
The practical sequence
- Scan first. Clear the machine-checkable failures in bulk — they are usually the largest raw count and the cheapest to fix. This is what the platform in these reviews is for.
- Then audit. Put a human on what is left, where the judgement calls and the legal exposure live.
- Then monitor. Accessibility regresses with every deploy. One reviewer's phrase for it, quoting the founder, is that this is not a set-it-and-forget-it project.
Common questions
- What is a manual accessibility audit?
- A human expert works through a site against the WCAG success criteria using a keyboard and a screen reader, judging things a scanner cannot evaluate — whether alt text is meaningful, whether focus order is logical, whether an error message actually tells the user how to fix the problem. The output is a report listing each failure, the criterion it breaches, its severity, and how to remediate it.
- Why is automated accessibility testing not enough on its own?
- Automated tools reliably detect machine-checkable failures such as a missing alt attribute, an unlabelled form field, or insufficient colour contrast. They cannot judge quality or intent. A scanner sees that an image has alt text; only a person can tell whether that alt text describes the image. A large share of WCAG success criteria require human judgement, so a clean automated scan is not the same as conformance.
- What does an automated scan catch reliably?
- Missing alt attributes, missing form labels, colour contrast below threshold, missing document language, duplicate IDs, empty links and buttons, missing table headers, and ARIA attributes that are invalid or misapplied. This is real work and it is the majority of raw issue count on most sites — it is simply not all of conformance.
- When do I need a manual audit rather than a scan?
- When you have a legal deadline or a procurement requirement, when you need an Accessibility Conformance Report or VPAT, when you are responding to a demand letter or complaint, or when your site has custom interactive components — a bespoke date picker, a multi-step form, a modal flow, a data grid — that a scanner cannot meaningfully evaluate.
- What standard should the audit be against?
- WCAG 2.2 Level AA is the common target. Section 508 references WCAG 2.0 AA for US federal procurement; the DOJ's ADA Title II rule sets WCAG 2.1 AA for state and local government entities; EN 301 549 governs the EU. An audit should state which version and level it tested against.
- Do the reviews on this site cover the manual audit service?
- No. The AppSumo reviews collected here are about the self-serve platform — the scanner, the automatic fixes and the widget. None of them evaluate an expert manual audit engagement, so this page makes no claims about that service on the strength of the reviews.