fix(executor): Fix compatibility issue when selfLink is no longer populated for k8s>=1.21. Fixes #6045

[terrytangyuan]

See context in #6008. Also Fixes #6045.

Checklist:

• My organization is added to USERS.md.

Tips:

• Your PR needs to pass the required checks before it can be approved. If the check is not required (e.g. E2E tests) it does not need to pass
• Sign-off your commits to pass the DCO check: git commit --signoff.
• Run make pre-commit -B to fix codegen or lint problems.
• Say how how you tested your changes. If you changed the UI, attach screenshots.

[codecov]

Codecov Report

Merging #6014 (1cc1b86) into master (f1fcb43) will increase coverage by 0.01%.
The diff coverage is 81.81%.

@@            Coverage Diff             @@
##           master    #6014      +/-   ##
==========================================
+ Coverage   47.52%   47.53%   +0.01%     
==========================================
  Files         248      248              
  Lines       15721    15730       +9     
==========================================
+ Hits         7472     7478       +6     
- Misses       7312     7316       +4     
+ Partials      937      936       -1     

Impacted Files	Coverage Δ
workflow/executor/resource.go	28.96% <81.81%> (+3.67%)	⬆️
workflow/metrics/server.go	12.50% <0.00%> (-4.17%)	⬇️
cmd/argo/commands/get.go	56.45% <0.00%> (-0.65%)	⬇️
workflow/controller/operator.go	71.02% <0.00%> (+0.05%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f1fcb43...1cc1b86. Read the comment docs.

[terrytangyuan]

@alexec Only tests for a few of the executors failed and the errors are all different. Is there anything special about the CI environment for different executors (e.g. permissions) that I should be aware of?

The following log is from build for emissary executor:

logs | k8s-resource-tmpl-with-wf-m6jjx main Error: signal: terminated
1050
 controller | time="2021-05-26T04:37:38.961Z" level=info msg="Create pods/exec 500"
1051
 controller | time="2021-05-26T04:37:38.962Z" level=warning msg="failed to clean-up pod" action=terminateContainers error= key=argo/k8s-resource-tmpl-with-wf-m6jjx/terminateContainers

Different error in PNS executor:

 controller | time="2021-05-26T04:38:32.144Z" level=info msg="Pod failed: Error (exit code 143)" displayName=k8s-resource-tmpl-with-wf-hnk9z namespace=argo pod=k8s-resource-tmpl-with-wf-hnk9z templateName=main workflow=k8s-resource-tmpl-with-wf-hnk9z

[alexec]

Step 1. Let's re-run the jobs.

[terrytangyuan]

Okay tried a couple of things. It looks like each time different executors failed so there maybe something unstable hidden in the CI environment itself. Also the failure is unrelated to the change but rather the new test case itself: https://github.com/argoproj/argo-workflows/actions/runs/880489742

[terrytangyuan]

I've located the root cause which will be fixed in #6033. I'll clean up this PR once #6033 is merged so this PR can focus on supporting k8s>=1.21.

[terrytangyuan]

This PR has been rebased after #6033 got merged. @alexec Feel free to review whenever you get a chance.

[alexec]

Lets hold this for v3.2

[christopheblin]

@alexec

Lets hold this for v3.2

Without this fix, "workflow-of-workflows" is not working with k8sapi executor under "new" k8s versions (basically, I think any resource creation that takes a bit of time is not working, so it means Workflows but also Deployments)

Can you explain why the fix is hold ? Will it be available in latest or in a 3.1.1 soon ?

Or is there a workaround to "wait" the resource in a next step ?

[alexec]

Are we saying that self link issue is already impacting production systems? I.e. 1.21 is now running in production?

[christopheblin]

Thanks for the merge @alexec 👍

I am running Azure Kubernetes Services (AKS) which, by the analysis of logs by @terrytangyuan in #6045, has the selfLink issue

I'll try to test the "latest" image tomorrow (CET) and I'll come back with a new issue if needed