Caching of failed responses

[elsand]

Description

We're returning null/empty responses when HttpClient calls fail, causing these data to be cached. See discussion.

Reproduction

All infrastructure clients used for lookups (with the possible exception ofOrganizationRegistryClient) will return null or empty results upon any intermittent error. FusionCache will then happily cache those responses as valid results, and API callers won't be able to distinguish actual empty results with intermittent upstream failure.

Expected behavior

If the remote server is down (ie. we get some sort of socket/connection error), non-successful response codes or the returned data fails to unserialize to the expected model, an exception should be thrown. This will be caught by the FailSafe mechanism in FusionCache, which will reuse the stale data and attempt a new request after FailSafeThrottleDuration has elapsed. The actual request should use Polly with a retry (and perhaps backoff).

If no stale data is available, or if FailSafeMaxDuration is reached, the exception should bubble up and result in a 502 Bad Gateway to indicate to clients that something upstream is down. Exceptions should be logged / alerted as normally.

Actual behavior

No exceptions are thrown, and misbehaving upstream services cause empty/null results to be returned, which might cause confusion ("why is my inbox empty all of a sudden?")