fix: change fatal to panic.

[shuangkun]

comments here: #12817 (comment)

Motivation

Modifications

Verification

[shuangkun]

@agilgur5 could you help have a look for this, thanks!

[agilgur5]

some small changes -- we can use log.Panic actually

I also renamed this to a fix since Fatal / os.Exit is normally not recoverable; so these cases probably should use panic

[tczhao]

Up for a discussion,

This util FormulateResubmitWorkflow function
is only run by the argo servers,
I'm thinking maybe we should remove and replace all panic/fatal in this function with error/warn
so the server keeps alive and returns the appropriate status code,err

[agilgur5]

is only run by the argo servers,
I'm thinking maybe we should remove and replace all panic/fatal in this function with error/warn
so the server keeps alive and returns the appropriate status code,err

Yes, that's why I suggested the panic vs. fatal. Panics are recoverable and we have a few recover statements and functions.

Regarding the status code, these would generally all be 500s; they're generally indicative of a bug. I would think a panic would end up returning a 500, but I'm not sure

[shuangkun]

Up for a discussion,

This util FormulateResubmitWorkflow function is only run by the argo servers, I'm thinking maybe we should remove and replace all panic/fatal in this function with error/warn so the server keeps alive and returns the appropriate status code,err

Ok, changed fatal to error in FormulateResubmitWorkflow

[tczhao]

Yes, that's why I suggested the panic vs. fatal. Panics are recoverable and we have a few recover statements and functions.

right, found recover https://github.com/argoproj/argo-workflows/blob/main/util/grpc/interceptor.go#L21

[agilgur5]

LGTM, thanks for fixing these cases

[agilgur5]

@tczhao for reference all these node not found errors were added in #11451 as there were some actual bugs where nodes didn't exist but silently returned fine (because of how maps work in Go).

in the cases where we know for sure these errors will never happen, we should ignore the errors and not handle them at all, similar to what I did in #12917 .

the problem is that we don't know all the cases 😕
eventually these should all really be removed or only exist in a few places where, for instance, the status got corrupted and is not actually in the Controller's ability to fix, like if a user literally re-wrote the status with kubectl (or some other wacky race condition)

cc @isubasinghe for reference who did the most work on this (and I may have the second most context on it nowadays?)

[tczhao]

@agilgur5 Got it, thank you for the context

[agilgur5]

Backported cleanly to release-3.5 as 9c2581a