Expected 404 errors should not be reported as a failure

[icaroNZ]

For long-time operations, the endpoint returns a location header and is up to the backend to pull the endpoint for the response, the endpoint should return 404 until the request is processed and ready.
In K6 I was able to code my way around this:

 do {
    count++;
    result = postman[Request](name);
    if (!resultOk.includes(result.status.toString())) sleep(sleepDuration);
  } while (!resultOk.includes(result.status.toString()) && count < numberOfRetries);
  check(count, {
    "reties exceed maximum": count => count < numberOfRetries
  });

However in LoadImpact I receive this message:

The total number of HTTP(s) errors is higher than 15% during the first 100 completed script iterations.

We should somehow be able to tell LoadImpact when the 404 result is an expected and leave the error to be raised by the code in that case, so it will not clog the logs.

[na--]

Thanks for posting this issue! It's a very valid use case, and we've seen other similar issues that are even more unavoidable 😞 Whereas your use case might also be solved if we implement something like #884 (comment) (i.e. silence all metrics in a code region), others definitely need a way to tag non-2xx response statuses (or maybe even other non-HTTP errors like network/DNS/etc. failure?) as expected. Examples:

• When you use digest/NTLM HTTP authentication, it's expected that the remote server would reply with a 403 response to your initial request, so that you get the nonce/challenge/etc. you need to successfully authenticate. This is totally normal thing and is outside of the control of users, so we need some way to automatically mark/tag the metrics of that first response as "expected and normal status code", but not the second one. Connected issue: Refactor HTTP digest authentication #800
• Sometimes you're intentionally load testing and endpoint that is meant to return a 4xx/5xx status - you care about the metrics, but not about the status.

[imiric]

Hey, sorry for the long delay, we're only now starting work on the new HTTP API that aims to fix this.

But I'm curious: isn't this already fixed by http.expectedStatuses() and responseCallback? You could set responseCallback to expect 404s for specific requests, and k6 wouldn't report those responses as part of the http_req_failed metric.

I'm not sure how this would behave in the Cloud, especially after 3 years 😅 Maybe it's worth testing, otherwise we should close this as fixed. It would need special handling for digest auth, though, but maybe we should address that as part of #800.

[imiric]

This is fixed now, so I'll close this. We'll carry this feature over to the new API (initial design document).