Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

The fasthttp fs module does not provide a separate cache container for the Zstandard (zst) compression algorithm. #1827

Closed
newacorn opened this issue Aug 10, 2024 · 1 comment · Fixed by #1828

Comments

@newacorn
Copy link
Contributor

newacorn commented Aug 10, 2024

Issue Test

func TestAcceptEncodingIsOnlyZstd(t *testing.T) {
	const fileName = "test.txt"
	f, err := os.Create("/tmp/" + fileName)
	if err != nil {
		t.Fatal(err)
	}
	defer func() {
		os.Remove("/tmp/" + fileName)
	}()
	_, err = f.Write(bytes.Repeat([]byte("1"), 1000))
	if err != nil {
		t.Fatal(err)
	}
	f.Close()
	fs := FS{Root: "/tmp", Compress: true, CacheDuration: time.Hour * 20}
	h := fs.NewRequestHandler()
	server := Server{
		Handler: h,
	}
	pcs := fasthttputil.NewPipeConns()
	cliCon, serCon := pcs.Conn1(), pcs.Conn2()
	go func() {
		defer func() {
			_ = cliCon.Close()
		}()
		req := AcquireRequest()
		resp := AcquireResponse()
		defer func() {
			ReleaseRequest(req)
			ReleaseResponse(resp)
		}()
		req.SetRequestURI("http://127.0.0.1:7070/" + fileName)
		req.Header.Set(HeaderAcceptEncoding, "zstd")
		_, err = req.WriteTo(cliCon)
		if err != nil {
			t.Error(err)
			return
		}
		err = resp.Read(bufio.NewReader(cliCon))
		if err != nil {
			t.Error(err)
			return
		}
		if !bytes.Equal(resp.Header.ContentEncoding(), []byte("zstd")) {
			t.Error("not zstd encoding")
			return
		}
		resp.Reset()
		req.Header.Del(HeaderAcceptEncoding)
		_, err = req.WriteTo(cliCon)
		if err != nil {
			t.Error(err)
			return
		}
		err = resp.Read(bufio.NewReader(cliCon))
		if err != nil {
			t.Error(err)
			return
		}
		if len(resp.Header.ContentEncoding()) != 0 {
			t.Error("has content encoding header")
			return
		}
		if utf8.Valid(resp.Body()) != true {
			t.Error("invalid utf8 bytes sequence")
			return
		}
		err = cliCon.Close()
		if err != nil {
			t.Error(err)
		}
	}()
	err = server.ServeConn(serCon)
	if err != nil {
		t.Fatal(err)
	}

Result

=== RUN   TestAcceptEncodingIsOnlyZstd
    fs_fs_test.go:787: invalid utf8 bytes sequence
--- FAIL: TestAcceptEncodingIsOnlyZstd (0.00s)

Let's simply add a separate cache container for zstd.

@erikdubbelboer
Copy link
Collaborator

Good one, I'm open for a pull request.

newacorn added a commit to newacorn/fasthttp that referenced this issue Aug 11, 2024
Add a dedicated cache container for the zstd compression algorithm to prevent discrepancies between the response content and the implied Content-Encoding in certain scenarios.Fix: valyala#1827 (comment)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants