gateway-conformance
is a tool designed to test if an IPFS Gateway implementation complies with the IPFS Gateway Specification correctly. The tool is distributed as a Docker image, as well as a GitHub Action(s).
The test
command is the main command of the tool. It is used to test a given IPFS Gateway implementation against the IPFS Gateway Specification.
Input | Availability | Description | Default |
---|---|---|---|
gateway-url | Both | The URL of the IPFS Gateway implementation to be tested. | http://localhost:8080 |
subdomain-url | Both | The Subdomain URL of the IPFS Gateway implementation to be tested. | http://example.com |
json | Both | The path where the JSON test report should be generated. | ./report.json |
xml | GitHub Action | The path where the JUnit XML test report should be generated. | ./report.xml |
html | GitHub Action | The path where the one-page HTML test report should be generated. | ./report.html |
markdown | GitHub Action | The path where the summary Markdown test report should be generated. | ./report.md |
specs | Both | A comma-separated list of specs to be tested. Accepts a spec (test only this spec), a +spec (test also this immature spec), or a -spec (do not test this mature spec). | Mature specs only |
args | Both | [DANGER] The args input allows you to pass custom, free-text arguments directly to the Go test command that the tool employs to execute tests. |
N/A |
By default, only mature specs (reliable, stable, or permanent) will be tested if this input is not provided. You can specify particular specs without any prefixes (e.g., subdomain-gateway, trustless-gateway, path-gateway) to test exclusively those, irrespective of their maturity status.
To selectively enable or disable specs based on their maturity, use the "+" and "-" prefixes. Adding a "+" prefix (e.g., +subdomain-gateway) means that the spec should be included in the test, in addition to the mature specs. Conversely, using a "-" prefix (e.g., -subdomain-gateway) means that the spec should be excluded from the test, even if it is mature.
If you provide a list containing both prefixed and unprefixed specs, the prefixed specs will be ignored. It is advisable to use either prefixed or unprefixed specs, but not both. However, you can include specs with both "+" and "-" prefixes in the same list.
This input should be used sparingly and with caution, as it involves interacting with the underlying internal processes, which may be subject to changes. It is recommended to use the args
input only when you have a deep understanding of the tool's inner workings and need to fine-tune the testing process. Users should be mindful of the potential risks associated with using this input.
The subdomain-url
parameter is utilized when testing subdomain support in your IPFS gateway. It can be set to any domain that your gateway permits.
During testing, the suite keeps connecting to the gateway-url
while employing HTTP techniques to simulate requests as if they were sent to the subdomain.
This approach enables testing of local gateways during development or continuous integration (CI) scenarios.
A few examples:
Use Case | gateway-url | subdomain-url |
---|---|---|
CI & Dev | http://127.0.0.1:8080 | http://example.com |
Production | https://dweb.link | https://dweb.link |
- name: Run gateway-conformance tests
uses: ipfs/gateway-conformance/.github/actions/test@v1
with:
gateway-url: http://localhost:8080
specs: +subdomain-gateway,-path-gateway
json: report.json
xml: report.xml
markdown: report.md
html: report.html
args: -timeout 30m
docker run --network host -v "${PWD}:/workspace" -w "/workspace" ghcr.io/ipfs/gateway-conformance test --gateway-url http://localhost:8080 --json report.json --specs +subdomain-gateway,-path-gateway -- -timeout 30m
The extract-fixtures
command is used to extract the test fixtures from the gateway-conformance
tool.
Input | Availability | Description | Default |
---|---|---|---|
output | Both | The path where the test fixtures should be extracted. | ./fixtures |
merged | Both | Whether the fixtures should be merged into a single CAR file. | false |
- name: Extract gateway-conformance fixtures
uses: ipfs/gateway-conformance/.github/actions/extract-fixtures@v1
with:
output: fixtures
merged: false
docker run -v "${PWD}:/workspace" -w "/workspace" ghcr.io/ipfs/gateway-conformance extract-fixtures --output fixtures --merged false
You can find the workflow that runs the gateway conformance test suite against Kubo in the file .github/workflows/test.yml. This can serve as a good starting point when setting up your own test suite.
We've also aimed to keep the kubo-config.example.sh script and the Makefile as straightforward as possible to provide useful examples to get started.
We make minimal assumptions about the capabilities of the gateway being tested. Which means that we don't require nor expect the gateway to be writable. Therefore, you need to provision the gateway with test fixtures before running the test suite.
These fixtures are located in the ./fixtures
folders. We distribute tools for extracting them. Refer to the documentation for the extract-fixtures
command for more details.
Fixtures:
- Blocks & Dags: These are served as CAR file(s).
- IPNS Records: These are distributed as files containing IPNS Record serialized as protobuf. The file name includes the Multihash of the public key (IPNS Name) in this format:
pubkey(_optional_suffix)?.ipns-record
. We may decide to share CAR files in the future. - DNS Links: These are distributed as
yml
configurations. You can use the--merge
option to generate a consolidated.json
file, which can be more convenient for use in a shell script.
The examples are going to use gateway-conformance
as a wrapper over docker run -v "${PWD}:/workspace" -w "/workspace" ghcr.io/ipfs/gateway-conformance
for simplicity.
By default, all mature tests are run. Mature tests generally refer to specifications whose status is mature.
gateway-conformance test
gateway-conformance test --specs subdomain-gateway,path-gateway
gateway-conformance test --specs +subdomain-gateway
gateway-conformance test --specs -subdomain-gateway,-dnslink-gateway
gateway-conformance test --specs trustless-gateway,-trustless-gateway-ipns
Tests are skipped using Go's standard syntax:
gateway-conformance test -skip 'TestGatewayCar/GET_response_for_application/vnd.ipld.car/Header_Content-Length'
gateway-conformance extract-fixtures
gateway-conformance extract-fixtures --merged true
golang's default string formating package is similar to C. Format strings might look like "this is a %s"
where %s
is a verb that will be replaced at runtime.
These verbs collides with URL-escaping a lot, strings like /ipfs/Qm.../%c4%85/%c4%99
might trigger weird errors. We implemented a minimal templating library that is used almost everywhere in the test.
It uses {{name}}
as a replacement for %s
. Other verbs are not supported.
Fmt("{{action}} the {{target}}", "pet", "cat") // => "pet the cat"
Backticks enable use of verbatim strings, without having to deal with golang-specific escaping of things like double quotes:
Fmt(`Etag: W/"{{etag-value}}"`, "weak-key") // => "ETag: W/\"weak-key\""
It is required to always provide a meaningful {{name}}
:
Fmt(`/ipfs/{{cid}}/%c4%85/%c4%99`, fixture.myCID) // => "/ipfs/Qm..../%c4%85/%c4%99"
Values are replaced in the order they are defined, and you may reuse named values
Fmt(`<a href="{{cid}}">{{label}}}</a><a href="{{cid}}/index.html">index</a>`, fixture.myCID, "Link Title!") // => '<a href="Qm...">Link Title!</a><a href="Qm..../index.html">index</a>'
You may escape {{}}
by using more than two opening or closing braces,
Fmt("{foo}") // => "{foo}"
Fmt("{{{foo}}}") // => "{{foo}}"
Fmt("{{{{foo}}}}") // => "{{{foo}}}"
Fmt("{{{foo}}}") // => {{foo}}
This templating is used almost everywhere in the test sugar, for example in request Path:
Request().Path("ipfs/{{cid}}", myCid) // will use "ipfs/Qm...."
The tool can generate XML, HTML and Markdown reports when used as a GitHub Action. However, when using the tool as a Docker container, you can generate these reports by using the saxon
Docker image. You can draw inspiration from the gotest-json-to-junit-xml and the junit-xml-to-html GitHub Actions.
Please let us know if you would like to see this feature implemented directly in the Docker image distribution.
- How to deal with subdomains & configuration (t0114 for example)?
- Some test relies on querying URLs like
http://$CIDv1.ipfs.example.com/
. Whilehttp://$CIDv1.ipfs.localhost/
works by default, do we need / want to test with.example.com
?
- Some test relies on querying URLs like
- Debug logging
- Set the environment variable
GOLOG_LOG_LEVEL="conformance=debug"
to toggle debug logging.
- Set the environment variable