testing: use Hurl in CI to test Caddy against spec

[mohammed90]

Since #5704 was posted, we've been on-and-off brainstorming how to approach testing of a web server. We sorta agreed that declarative approach is desired but weren't aware of any tools that'd facilitate the declarative approach nor had a concrete plan. We just knew we need solid tests.

I have recently come across Hurl (https://github.com/Orange-OpenSource/hurl) and was curious if it meets our needs. It is declarative. It makes HTTP calls. It stands on shoulders of The Giant®, namely curl. The PoC presented in this branch seems to work. In fact, the PR #6249 is a fix for a bug found while building this PoC.

This PR is to discuss the approach and to collaboratively add the tests. The core idea is simple:

HTTP handlers claim to conform to a particular behavior. The behavior can be specified through a collection of Hurl tests, i.e. a collection of HTTP requests and responses. The Hurl file defines the Spec the handler shall meet.

TODO:

• Agree on the structure/placement of the spec files
• Write the spec files for all the existing handlers by inspecting the docs and the code

For TODO number 2, code coverage is a helpful tool. There's a way to extract execution coverage of the hurl tests†, but I haven't found a neat way to present it on GitHub PRs/Actions.

Based on the work done to resolve #5849 and the existence of the project REDbot, we can translate those expectations and rules into Hurl files.

† Using this article as guide: Build Caddy with coverage instrumentation using go build -cover. Run Caddy using the command GOCOVERDIR=./coverdir caddy run, then run the Hurl tests. Stop Caddy with either caddy stop or Ctrl-C. Run go tool covdata textfmt -i=coverdir -o profile.txt. Run go tool cover -html profile.txt. An HTML page is opened in the browser with each file annotated by color for whether it was executed or not.

[mohammed90]

Example of how it looks in the Actions run summary:

Example of how failure is presented:

Example of how success is presented:

[github-actions]

Test Results

6 tests   6 ✅  2s ⏱️
6 suites  0 💤
1 files    0 ❌

Results for commit c718744.

♻️ This comment has been updated with latest results.

[mholt]

Ooo, I like where this is going! Will revisit this after 2.8.

[dkarlovi]

@mohammed90 this looks great! \o/

I'm only wondering one thing:

we can translate those expectations and rules into Hurl files

Since the expectations are basically "the HTTP protocol" (and related stuff), would Caddy actually be the right place to do this? RedBot's value here IMO is exactly that it already knows what requests need to be done and what assertions need to be done against them, converting that into Hurl-based specs seems like it would be a sizeable project with a sizeable deliverable.

Would creating this "Hurl-based HTTP spec test suite" be better as a standalone project which Caddy (and others, hopefully) can take advantage of and, hopefully, maintain?

[mohammed90]

Since the expectations are basically "the HTTP protocol" (and related stuff), would Caddy actually be the right place to do this? RedBot's value here IMO is exactly that it already knows what requests need to be done and what assertions need to be done against them, converting that into Hurl-based specs seems like it would be a sizeable project with a sizeable deliverable.

Caddy can be the right place :) we're aiming to test Caddy conformance to the spec. I skimmed the REDbot repo, and it isn't too complex. Integrating REDbot itself into the CI pipeline might be more of a hassle to maintain. Translating the behavior into Hurl files makes the expectations easier to understand and poke.

Would creating this "Hurl-based HTTP spec test suite" be better as a standalone project which Caddy (and others, hopefully) can take advantage of and, hopefully, maintain?

Perhaps, but maintaining such project is beyond my capacity. I can't initiate and commit to it (my personal backlog is too long already). I may help if it's maintained by a group.