Like so many other accessibility tools, such as the Accessibility Developer Tools by Google, aXe by Deque, and HTML_CodeSniffer by Squiz to name a few, our proprietary accessibility conformance testing engine at Siteimprove runs within the context of a browser. The reason why this seems to be the de facto way of implementing an accessibility tool is obvious: The browser is the tool used to consume your website, so why not test directly within that very tool? Through the APIs exposed by the browser, we get access to all the information needed in order to assess the accessibility of a website; the structure we can access and inspect through the DOM, information about styling can be gained through the CSSOM, and soon we also get our hands on a standardised accessibility tree through the AOM.
However, not all is good in the land of browsers. Rendering a website is an inherently non-deterministic process and the timing of network requests, script execution, the content of request headers, and much more, all play a role in what the final result will look like. In most cases, this will directly affect the assessment of a tool that runs within the browser and will become very apparent at scale. At Siteimprove, we feel the effect of this on a daily basis; a customer asking us why we came up with a certain result and us having little to no clue because we cannot replicate the exact circumstances that led to that result. This is a frustrating experience for both our customers and ourselves as it makes it difficult to reason about our tool.
We want to fix this and we want to fix it for good. To do so, we must ensure that we have the ability to exactly replicate the results of a given accessibility assessment. Ideally, as many unknown browser variables as possible should be taken out of the equation and the browser only be used for what is absolutely necessary.
We will abandon any sort of dynamic analysis within the context of a browser. The input to Alfa will be static data and any assessment must be made based on that data alone. A browser may or may not be involved in the construction of the data, but the browser will not be required for any further assessment thereof.
If additional data is needed by a given accessibility rule, we will adjust the data format to meet the needs of the rule. We will also carefully consider the extent of the data format as to not bloat it with information that could otherwise be inferred from existing data. Ideally, the size of the data when serialised and stored on disk will not be much larger than the size of the original source code on which the data is based.
Accepted.
By moving from dynamic analysis performed within the context of a browser towards static analysis potentially performed outside the context of a browser, we gain the ability to exactly replicate the results of a given accessibility assessment as the assessment will be based on static data alone. If we wish to know why we came up with a certain result, we therefore simply inspect the data on which the result is based. While the construction of the data might still be a non-deterministic process if a browser is involved, the accessibility assessment will be entirely deterministic.
A side effect of operating solely on static data is that we can no longer rely on many of the APIs exposed by the browser, such as those used for DOM traversal and property access. These APIs will therefore have to be implemented by Alfa itself, which in many cases will be a non-trivial amount of work.