feat: Add webHDFS support. Fixes #7540

[alexdittmann]

Signed-off-by: Alexander Dittmann alexander.dittmann@sap.com

Fixes #7540

This PR adds support for webHDFS input/output artifacts, supporting two ways for authentication:

• via certificates: the client authenticates at the webHDFS endpoint via a tls certificate pair. Here, the user has to provide the certificate pair to authenticate with.
• via OAuth2: the client authenticates at the webHDFS endpoint via an OAuth2 token. Here, the user has to provide a client_id, client_secret and a tokenURL.
  An example yaml can be found here: https://github.com/argoproj/argo-workflows/compare/master...alexdittmann:feat-add-webhdfs-support?expand=1#diff-36d89ba256bdc0e7c8728a206b718b821dd7e38db2043abb699290d1204be7bb

Note:
I initially created a (private) fork of this repo, where I successfully tested these changes against, both with an Azure and SAP Hana Datalake store. When creating this official fork from the latest master, however, submitting new workflows always resulted in StartError of the workflow pod. Here is the events log:

Events:
  Type     Reason     Age   From               Message
  ----     ------     ----  ----               -------
  Normal   Scheduled  21s   default-scheduler  Successfully assigned argo/awesome-poochenheimer to k3d-k3s-default-server-0
  Normal   Pulled     22s   kubelet            Container image "quay.io/argoproj/argoexec:latest" already present on machine
  Normal   Created    22s   kubelet            Created container init
  Normal   Started    22s   kubelet            Started container init
  Normal   Pulled     22s   kubelet            Container image "quay.io/argoproj/argoexec:latest" already present on machine
  Normal   Created    21s   kubelet            Created container wait
  Normal   Started    21s   kubelet            Started container wait
  Normal   Pulled     21s   kubelet            Container image "argoproj/argosay:v2" already present on machine
  Normal   Created    21s   kubelet            Created container main
  Warning  Failed     21s   kubelet            Error: failed to create containerd task: failed to create shim: OCI runtime create failed: container_linux.go:380: starting container process caused: exec: "/var/run/argo/argoexec": stat /var/run/argo/argoexec: no such file or directory: unknown

Im using k3d btw. Can this error somehow be mitigated? (I have run make argo-exec already)

Also appreciate any feedback!

Thanks
Alex

[alexec]

I have a question. How is this different to using a HTTP artifact? Could we extend HTTP artifact somehow to support TLS client cert + oauth2? That would give HTTP users lots more features, but also work for WebHDFS.

[alexdittmann]

I have a question. How is this different to using a HTTP artifact? Could we extend HTTP artifact somehow to support TLS client cert + oauth2? That would give HTTP users lots more features, but also work for WebHDFS.

When I initially started working, the HTTP artifact only had the basic input artifact. Since webhdfs is a strict protocol (e.g. clearly defined how data is pushed), i thought it would be easier to handle it separately. However, taking a look at the recent changes to the http artifact, i see that that it would fit quite well into there now. The only thing that would need to be taken care of is the redirect logic.

[alexec]

I thought that might be the case. HTTP only change this week to be amenable to this, so your timing is perfect.

We should probably add an example of WebHDFS to the docs.

[alexdittmann]

closing this PR in favour for a new one: #8468