The Accessibility Audit
So far in the narrative, Greytower has not been involved. Our task was to, during January and February of 2003, check all 92 sites against the WCAG 1.0 checkpoints.
The audits were limited in scope: the result was going to show a
fever curve
of government agency websites. Details were omitted
[1],
and for each checkpoint the answer yes, no or
n/a (not applicable) would be returned.
To do this job, Greytower devised a methodology which is very close to the WAI Preliminary Review, yet not exactly the same. The major deviations are:
- All Greytower audits starts with the establishment of a baseline. This was simple, as it was more or less sorted. We did add five checkpoints not included in the WCAG for a total of 70 checkpoints.
- The automated testing tool used is an in-house product called siteSifter. The reasons for using this tool is partially financial, partially for efficiency [2], and partially for the way reports are produced.
The efficiency factor came into play in particular regarding the time-constraints, as all 92 sites were to be analyzed in 30 calendar days. Given weekends this meant that we had to process up to 4 sites per working day. A daunting task indeed.
Machinations
To start with, we mirrored 92 websites, limiting ourselves to a maximum of 2,500 documents from each. Not all had that many, but the total number of documents retrieved ended up at 37,462. Only HTML, CSS and Javascript files were collected.
Mirrors created, siteSifter was set to process each site and create a report. Various format reports can be created; the one chosen is a summary style as shown in appendix A.
Analysis
Analyzing the summary report from siteSifter takes abit of practice, and I won't go into the details of the process here. Sufficient to say that this report then form the basis for reading source-code, using a variety of user-agents [3] to double-check siteSifter results and evaluate those checkpoints the tool cannot control.
For each site, a report was written up for the Audit Office. A short description of the overall accessibility where added per site. The results were also added to a database maintained by Pollux Feedback AB [4]
Do it again ?
My initial reaction, after the fact, was No way!
. With time, that
has mellowed into a more cautious If well paid
. There is no doubt
in my mind that this review has been a valuable one. Swedish Government
Agencies can learn much from it.
However, the work was not without difficulty, nor are the results without flaws. The entire job was done in far too little time for comprehensive analysis, something which has been repeatedly pointed out by our most fierce competitor on the Swedish market.
Their reasons for claiming that the test results show a worse result than is actually the case is, and will remain, their own. Personally I feel that the overall picture of accessibility on these websites are correct.
The work needs repeating, however. The difference between "one point" and "any point" [5] accessibility can not be too strongly emphasized when working with Government websites. A yearly audit would be the minimum recommended.
1
On details: these were by necessity omitted. Doing a complete accessibility
audit - including
WCAG compliancy,
consequence analysis, and suggested/recommended corrections to code,
structure and organization - would, for 92 websites, take around 230
working days, or 46 weeks. Including vacations and other details this
amounts to about a year of work on the audits alone. The cost would be
staggering.
2
On efficiency: very few tools can efficiently scan 2,500 documents in a
file tree for accessibility issues and come up with a summary which can
easily be included in other reporting tools. siteSifter is designed to do
just that.
3
On user-agents: Mozilla, Netscape, Lynx, and Internet Explorer. JAWS was
not available to us during these tests.
4
On Pollux: ironically, the web forms used for entering data were quite
inaccessible, violating in particular checkpoint 6.3. Using a graphical
browser capable of Javascript slowed the work down considerably.
5
This is the difference between saying "At ONE point this website was
accessible to all", and "This website is accessible to all at ANY point".