feat: Add WebHDFS support for HTTP artifacts. Fixes #7540

[alexdittmann]

Signed-off-by: Alexander Dittmann alexander.dittmann@sap.com

Fixes #7540

This PR adds support for webHDFS for HTTP input/output artifacts. This is accomplished by:

• explicitly following temporary redirects (status code 307, see here)
• adding supporting for two ways of authentication:
  • via certificates: the client authenticates via a tls certificate pair
  • via OAuth2: the client authenticates via an OAuth2 token

An example yaml can be found here: https://github.com/argoproj/argo-workflows/compare/master...alexdittmann:feat-webhdfs-support-for-http-art?expand=1#diff-731816132908abcbbf4286af6ac8c6d9241ef1d7de7f6f00d92a6416e8106e8f

[alexdittmann]

What is the procedure with the required checks before making the PR ready for review, is there some action required? Three of them passed, but the others are in the "Expected — Waiting for status to be reported".

And another thing regarding updates of the PR: Do I always have to make sure that there is only one single commit, or is this only required for the PR creation?
Thanks so far!

[alexec]

some minor change, I think we probably want to have an E2E test for HTTP, can you check? if not, can you add a http_test.go, you might be able use the minio to test against.

[alexdittmann]

some minor change, I think we probably want to have an E2E test for HTTP, can you check? if not, can you add a http_test.go, you might be able use the minio to test against.

First of all, thanks for your feedback!
I agree, an e2e test would make sense, but I am not sure how to start. I was looking for an example and i only found something for s3 in the e2e/functional_test.go (e.g. here). Should I create something similar for http in a new http_test.go file or am i on the wrong track?

[alexec]

Yes. I would base it on ArtifactsSuite.

[alexec]

To write through tests, you'd need to start (I'm guessing) Hadoop. We do something similar for S3, by starting a single MinIO pod in the test system. MinIO is lightweight, so is Hadoop lightweight?

[alexdittmann]

Hi, thanks for the input. I would suggest using httpbin.org, this allows for testing the http artifacts also with the authentication types (only caveat is that this only works for the input artifacts, because there are only auth endpoints for GET).

• see https://github.com/argoproj/argo-workflows/pull/8468/files#diff-ad7c3d8804096865ffa2e3e2e2376865a3d9d6f3663b0d9f17ce09b498a72478 ff. for the deployment
• https://github.com/argoproj/argo-workflows/pull/8468/files#diff-2168186c551dc5494feb093ab608a3d3ffcd6baeb6f3e3b87e84b64506ec595b for the test suite
• and https://github.com/argoproj/argo-workflows/pull/8468/files#diff-25464b69c7df73d31747ab19e0ea3f6149ca0696615eda42b9d3283e0ff31d85 ff. for the test data
  What do you think?

[alexdittmann]

Can we make this ready for review and enable the tests? Or is there still something missing?

[alexec]

Just more minor comments. Because the PR is draft, I've only provided comments so far. One you mark it ready for review I'll check it out and run it myself and give a thorough review.

[alexec]

CI builds did not run. You also need to sync with master. Syncing should cause the builds to run.

[alexec]

We should have a doc (in ./docs) to show how to use this. E..g with Hadoop. This could be in this PR, or in another PR/issue.

#8635