Concurrency

Concurrent processing of documents is enabled with the TestFilesConcurrently config boolean. External link checking and to a much lesser extent, still significant on a large site, the parsing of HTML files occupies the most time in htmltest. Running these tasks concurrently speeds up testing.

Call Stack

As for sequential testing we start via the CLI

• cmd/main.go calls
• htmltest.Test sets up, parses the dir and then with a full DocumentStore calls
• htmltest.testDocuments which manages the goroutines that call
  • testDocument
  • testDocument
  • testDocument
  • ...

A channel is used in htmltest.testDocuments to control the number of testDocument goroutines started. Without this we hit the limit of open files pretty quickly on a big site.

The only other thing that's limited is external link checking (HTTP GETs), again with a channel stored in HtmlTest.httpChannel. Opening many HTTP connections causes them all to be slow, meaning the timeout has to be increased. It's not been tested but seems straightforward that lowering this limit will improve performance overall.

Current Issues

• #5 The cache is thread safe, however if the first set of documents (128) all ask for the same remote resource, a font for example, 128 HTTP calls are queued. The first cache get call to get through should set a STANDBY parameter on that item in the cache; creation of the cache ref should be made on this first GET. A mutex will be needed here to prevent multiple standbys getting through.