Query activity state when GetExecBatchResults call fails due to GSB error

[azawlocki]

An idea from @mfranciszkiewicz (from discord):

(...) Failing calls to the GetExecBatchResults endpoint are not unusual, but the cause is usually limited to "endpoint not registered". In this kind of situation, the requestor could try to query activity state, which would provide us with the following information:

• if it's not possible to query activity state, the provider is gone or having networking issues,
• if we can retrieve the state (most probably "Terminated"), we can log a message on premature activity termination on provider's side. Last known activity state is stored by the provider in their database for such cases.

The latter could be extended with state metadata / cause of the issue; this task already resides in core team's backlog.

My refinement:

I'm trying to figure out where call retrying fits in this picture, does the following look reasonable to you:

if call failed with "endpoint address not found":
    if try_query_activity_state() == "Terminated":
        log("Premature activity termination by provider")
    else:  # a different state or query failed
        repeat the call
else:  # call failed for other reason
    repeat the call

@mfranciszkiewicz

yes, that's basically it

edit: mfranciszkiewicz: change error message to endpoint address not found

[azawlocki]

Supersedes #584