feat: run kubo gateway sharness tests

[hacdias]

• Kubo running on unique port.
• Kubo gateway port exported as PROXY_GATEWAY_URL
• Bifrost gateway running with PROXY_GATEWAY_URL
• Sharness running against Bifrost
  • command line sharness
    • Skip writable gateway tests
    • Skip api-no-gateway
• See feat: allow gateway endpoint overwrite in tests ipfs/kubo#9700

[hacdias]

Test Failure Investigation (ONGOING)

272 179 129 105 103 tests seem to consistently fail on CI.

• t0109-gateway-web-_redirects.sh (2)
  • Reliance on Kubo Gateway.PublicGateways for DNSLink gateways.
  • DNSLink Gateway giving 404.
• t0112-gateway-cors.sh (6) tests
  • Reliance on Kubo's Gateway.Headers for custom headers.
  • Reliance on Kubo's API.HTTPHeaders for /api/v0.
    • /api/v0/cat now redirects to regular gateway, so different headers are also applied. Current tests do not follow rediretcs.
• t0115-gateway-dir-listing.sh (7)
  • DNSLink Gateway giving 404. Reliance on IPFS_NS_MAP?
• t0114-gateway-subdomains.sh (88)
  • Many 404s, likely caused by reliance on Kubo's configuration. Reliance on IPFS_NS_MAP?

Can fix all t0109-gateway-web-_redirects.sh, all t0115-gateway-dir-listing.sh, some (7) in t0114-gateway-subdomains.sh by hijacking the test-lib by, for example, creating a .patch file that we would apply in the CI here. It would also simplify other things. Having a .patch file could also help us implement something to go around the custom headers tests.

diff --git a/test/sharness/lib/test-lib.sh b/test/sharness/lib/test-lib.sh
index 3aecaec99..15b42cc07 100644
--- a/test/sharness/lib/test-lib.sh
+++ b/test/sharness/lib/test-lib.sh
@@ -287,6 +287,17 @@ test_launch_ipfs_daemon() {
     pollEndpoint -host=$API_MADDR -v -tout=1s -tries=60 2>poll_apierr > poll_apiout ||
     test_fsh cat actual_daemon || test_fsh cat daemon_err || test_fsh cat poll_apierr || test_fsh cat poll_apiout
   '
+
+  test_expect_success "start bifrost gateway" '
+    PROXY_GATEWAY_URL=http://$GWAY_ADDR KUBO_RPC_URL=http://$API_ADDR GOLOG_LOG_LEVEL="bifrost-gateway=debug" bifrost-gateway --gateway-port 9090 &
+    BIFROST_PID=$! &&
+
+    GWAY_MADDR=/ip4/127.0.0.1/tcp/9090 &&
+    GWAY_ADDR=$(convert_tcp_maddr $GWAY_MADDR) &&
+    GWAY_PORT=$(port_from_maddr $GWAY_MADDR) &&
+
+    pollEndpoint -host=$GWAY_MADDR -v -tout=1s -tries=60
+  '
 }
 
 test_launch_ipfs_daemon_without_network() {
@@ -354,6 +365,14 @@ test_kill_ipfs_daemon() {
   test_expect_success "'ipfs daemon' can be killed" '
     test_kill_repeat_10_sec $IPFS_PID
   '
+
+  test_expect_success "'bifrost-gateway' is still running" '
+    kill -0 $BIFROST_PID
+  '
+
+  test_expect_success "'bifrost-gateway' can be killed" '
+    test_kill_repeat_10_sec $BIFROST_PID
+  '
 }
 
 test_curl_resp_http_code() {

By making Bifrost Gateway follow Kubo's configuration during testing, we can further reduce the amount of failing tests (requires above patch):

diff --git a/handlers.go b/handlers.go
index 3bfe0b4..e559cb4 100644
--- a/handlers.go
+++ b/handlers.go
@@ -1,9 +1,12 @@
 package main
 
 import (
+	"encoding/json"
 	"fmt"
 	"math/rand"
 	"net/http"
+	"os"
+	"path/filepath"
 	"strconv"
 	"time"
 
@@ -40,6 +43,17 @@ func withRequestLogger(next http.Handler) http.Handler {
 	})
 }
 
+type kuboConfig struct {
+	API struct {
+		HTTPHeaders map[string][]string
+	}
+	Gateway struct {
+		HTTPHeaders    map[string][]string
+		NoDNSLink      bool
+		PublicGateways map[string]*gateway.Specification
+	}
+}
+
 func makeGatewayHandler(bs bstore.Blockstore, kuboRPC []string, port int, blockCacheSize int, cdns *cachedDNS) (*http.Server, error) {
 	// Sets up an exchange based on the given Block Store
 	exch, err := newExchange(bs)
@@ -69,17 +83,6 @@ func makeGatewayHandler(bs bstore.Blockstore, kuboRPC []string, port int, blockC
 	}
 
 	headers := map[string][]string{}
-	gateway.AddAccessControlHeaders(headers)
-
-	gwConf := gateway.Config{
-		Headers: headers,
-	}
-
-	gwHandler := gateway.NewHandler(gwConf, gwAPI)
-	mux := http.NewServeMux()
-	mux.Handle("/ipfs/", gwHandler)
-	mux.Handle("/ipns/", gwHandler)
-	mux.Handle("/api/v0/", newKuboRPCHandler(kuboRPC))
 
 	// Note: in the future we may want to make this more configurable.
 	noDNSLink := false
@@ -96,6 +99,57 @@ func makeGatewayHandler(bs bstore.Blockstore, kuboRPC []string, port int, blockC
 		},
 	}
 
+	// Shamelessly applying Kubo's configuration to the public gateways such that
+	// the sharness tests pass...
+	if os.Getenv("KUBO_SHARNESS") == "true" {
+		kuboPath := os.Getenv("IPFS_PATH")
+		kuboConfigPath := filepath.Join(kuboPath, "config")
+
+		v, err := os.ReadFile(kuboConfigPath)
+		if err != nil {
+			return nil, err
+		}
+
+		var cfg *kuboConfig
+		err = json.Unmarshal(v, &cfg)
+		if err != nil {
+			return nil, err
+		}
+
+		if cfg.Gateway.HTTPHeaders != nil {
+			headers = cfg.Gateway.HTTPHeaders
+		}
+
+		noDNSLink = cfg.Gateway.NoDNSLink
+		publicGateways = map[string]*gateway.Specification{
+			"localhost": {
+				Paths:         []string{"/ipfs/", "/ipns/", "/api/", "/p2p/"},
+				UseSubdomains: true,
+			},
+		}
+
+		for hostname, gw := range cfg.Gateway.PublicGateways {
+			if gw == nil {
+				delete(publicGateways, hostname)
+			} else {
+				publicGateways[hostname] = &gateway.Specification{
+					Paths:         gw.Paths,
+					NoDNSLink:     gw.NoDNSLink,
+					UseSubdomains: gw.UseSubdomains,
+					InlineDNSLink: gw.InlineDNSLink,
+				}
+			}
+		}
+	}
+
+	gateway.AddAccessControlHeaders(headers)
+	gwConf := gateway.Config{Headers: headers}
+	gwHandler := gateway.NewHandler(gwConf, gwAPI)
+	mux := http.NewServeMux()
+	mux.Handle("/ipfs/", gwHandler)
+	mux.Handle("/ipns/", gwHandler)
+	mux.Handle("/api/v0/", newKuboRPCHandler(kuboRPC))
+
 	// Creates metrics handler for total response size. Matches the same metrics
 	// from Kubo:
 	// https://github.com/ipfs/kubo/blob/e550d9e4761ea394357c413c02ade142c0dea88c/core/corehttp/metrics.go#L79-L152

[hacdias]

I'm really not sure if working on this is a good usage of time. I think focusing on having conformance tests at (https://github.com/ipfs/gateway-conformance) is much more useful at this point in time. Many of the current tests are tightly coupled with Kubo's internals, especially with the ipfs config command. I don't really have a good suggestion on how to handle those tests (many from the subdomains/dnslink gateways and cors).

[lidel]

Thank you for the investigation into this @hacdias.

I agree with you: reusing sharness here, while technically possible, is a big time sink that ends up adding more cruft to the codebase, making everything more brittle.

Let's not do this, and instead set up https://github.com/ipfs/gateway-conformance instead – let's continue in #63
Even if the coverage is not as good, we will be dogfooding it, and have more incentives to make it better.

ps. @hacdias mind opening a new PR that fixes bugs(?) fixed in routing.go and gateway.go ?