fix: Add logging to aid troubleshooting

[copierrj]

While investigating stability issues we discovered that Argo wasn't retrying various failing k8s api calls. We didn't really understand what was going on until we deployed an experimental Argo build containing additional logging.

Therefore we propose make the root cause of failures more obvious and indicate if an error is considered to be transient or not.

Checklist:

• My organization is added to USERS.md.

[codecov]

Codecov Report

Merging #5501 (e02f13b) into master (3065941) will increase coverage by 0.16%.
The diff coverage is 100.00%.

❗ Current head e02f13b differs from pull request most recent head be01721. Consider uploading reports for the commit be01721 to get more accurate results

@@            Coverage Diff             @@
##           master    #5501      +/-   ##
==========================================
+ Coverage   46.54%   46.70%   +0.16%     
==========================================
  Files         240      240              
  Lines       15001    14994       -7     
==========================================
+ Hits         6982     7003      +21     
+ Misses       7119     7093      -26     
+ Partials      900      898       -2     

Impacted Files	Coverage Δ
util/errors/errors.go	100.00% <100.00%> (ø)
pkg/apiclient/apiclient.go	5.55% <0.00%> (-4.45%)	⬇️
cmd/argo/commands/get.go	56.66% <0.00%> (ø)
pkg/apiclient/http1-client.go	0.00% <0.00%> (ø)
pkg/apiclient/logs-intermediary.go	0.00% <0.00%> (ø)
pkg/apiclient/http1/event-watch-client.go	0.00% <0.00%> (ø)
...lient/error-translating-workflow-service-client.go	0.00% <0.00%> (ø)
...o-kube-cluster-workflow-template-service-client.go	0.00% <0.00%> (ø)
workflow/controller/operator.go	70.47% <0.00%> (+0.21%)	⬆️
cmd/argoexec/commands/emissary.go	50.00% <0.00%> (+1.56%)	⬆️
... and 2 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 3065941...be01721. Read the comment docs.

[simster7]

Thanks! This could be very useful

[simster7]

Looks like we need to lint, then we should be good to go