add http zstd reader for cdi

[huangzynn]

add http zstd reader for cdi

Enhancement: Support for zstd added

[kubevirt-bot]

Hi @huangzynn. Thanks for your PR.

I'm waiting for a kubevirt member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[mhenriks]

/ok-to-test

[awels]

Hi, thanks for the PR, I would like to ask if you could add some functional tests to demonstrate the feature working. We deploy an http server in our CI that automatically generates a variety of formats that are then used in our tests. If you look at https://github.com/kubevirt/containerized-data-importer/blob/main/tools/cdi-func-test-file-host-init/main.go#L43-L53 it provides a list of different formats that need to be generated. The actual code that generates the different formats can be found here https://github.com/kubevirt/containerized-data-importer/blob/main/tests/utils/fileConversion.go
Once the appropriate code is added, this should generate zstd formatted files in the http server. And you can add a new datavolume test similar to: https://github.com/kubevirt/containerized-data-importer/blob/main/tests/datavolume_test.go#L555-L578 with a different URL that points to the new file you created above.

If you have issues with any of that feel free to contact me here or on slack, and I can walk you through it.

Other than that, it looks good and I have no doubt it works, I just want to be able to prove it to other people that ask.

[awels]

Also to sign your commits, you can do a git commit --amend -s and it should automatically sign it and make the DCO bot happy

[huangzynn]

/retest

[huangzynn]

/retest

[awels]

Almost there, one more thing, if you look at https://github.com/kubevirt/containerized-data-importer/blob/main/tests/datavolume_test.go#L579-L602 you will see it build a DataVolume that imports a plain qcow2 file. If you could add another entry that imports the .zstd file you created in the http server, so we can verify the entire flow works. All you have to do is modify the name to lets say dv-https-import-zstd and the url needs to point to the new file. And the bound message will have a different name and should be something like PVC dv-https-import-zstd Bound. You can run the test suite with make test-functional and you can focus on just that one test by modifying table.Entry to table.FEntry. Just make sure to remove the F once you are happy that the test is passing.

[awels]

So the test looks good, but it is failing because the http server doesn't contain the file we are looking for (the importer pod is returning a 404). Let me grab the PR and I can see what exactly is being generated when the http server starts. It is likely just a file name that needs to change or we don't generate for tiny core.

[huangzynn]

So the test looks good, but it is failing because the http server doesn't contain the file we are looking for (the importer pod is returning a 404). Let me grab the PR and I can see what exactly is being generated when the http server starts. It is likely just a file name that needs to change or we don't generate for tiny core.

Sorry, this is my code error.
e7c99c5

[huangzynn]

/retest

[awels]

Still the same 404, hopefully I will have some time today to take a look

[awels]

It was so silly, apparently the generated file name is tinyCore.iso.zst so we just have to fix the URL to point to the right name. I modified it locally and ran the test and it passed

[awels@awels containerized-data-importer]$ git diff
diff --git a/tests/utils/datavolume.go b/tests/utils/datavolume.go
index 95d4fa14d..18e7f0f39 100644
--- a/tests/utils/datavolume.go
+++ b/tests/utils/datavolume.go
@@ -48,7 +48,7 @@ const (
        // HTTPSTinyCoreQcow2URL provides a test (https) url for the tineyCore qcow2 image
        HTTPSTinyCoreQcow2URL = "https://cdi-file-host.%s/tinyCore.qcow2"
        // HTTPSTinyCoreZstURL provides a test (https) url for the tineyCore zst image
-       HTTPSTinyCoreZstURL = "https://cdi-file-host.%s/tinyCore.zst"
+       HTTPSTinyCoreZstURL = "https://cdi-file-host.%s/tinyCore.iso.zst"
        // TinyCoreQcow2URLRateLimit provides a test url for the tineyCore qcow2 image via rate-limiting proxy
        TinyCoreQcow2URLRateLimit = "http://cdi-file-host.%s:82/tinyCore.qcow2"
        // TinyCoreQcow2GzURLRateLimit provides a test url for the tineyCore qcow2.gz image via rate-limiting proxy

[huangzynn]

It was so silly, apparently the generated file name is tinyCore.iso.zst so we just have to fix the URL to point to the right name. I modified it locally and ran the test and it passed

Thank you for your help

[huangzynn]

/retest

[huangzynn]

/retest

[awels]

/approve
I will add a lgtm later today, want to give other people a chance to review.

[kubevirt-bot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: awels

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [awels]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[mhenriks]

/lgtm