Optional HTTP Request failure step down

[BenjamenMeyer]

Provide a configuration setting for HTTP requests to be able to scale back in response to specified error codes.

Use Case: I'm performance testing an API using locustio. However, when it reaches the limits it starts generating 504 errors and some 400 and 404 errors. The problem is that since the request fails quickly, the server gets hammered harder at this point.

It would be good if I could specify in the configuration something like:

# all times in seconds
scale_back = {
   504: [5, 10, 15, 30, 60, 120],
   404: [10, 30],
   401: [-1],  # never retry
   400: [5, 10],
}

Several points:

• Allowing different HTTP response codes to have different step downs would allow for APIs that have different behaviors. A 504 error for instance may be recoverable if given some time.
• Specifying -1 would terminate any further requests being made. This would have be to be logged appropriately (perhaps a special exception or something) to make it clear why the locust worker stopped.
• Uses a list of times (in seconds) allows the response to be scaled back. On the first failure it would wait the first entry, if it immediately failed again, it would wait the second entry, and so on; however, if it succeeded or the failure code changed it would restart the time list traversal.
• Default behaviour would be the current behaviour; this must be explicitly optional

I've looked at the code and I'm not sure where this would get applied. The request is being made in client.request() but it would seem best to be applying this to wherever that function returns, which is not clear to me at the moment.

[cgoldberg]

I am -1 on this feature.
once you have reached capacity limits, I can't think of a case where you would want to continue running at a lower rate.

[BenjamenMeyer]

@cgoldberg it's not always about capacity of the API being tested, and even if you hit something that looks like capacity, you might want other requests to be able to complete.

For instance, suppose the API is using an external auth service; if the locust test hits a 401/403 due to issues with the auth serviec and then starts hammering the API harder it may cause other tests that did not get the 401/403 to fail. it's already known that the 401/403 caused a failure, but why should it make the other tests fail too?

Or for instance, if the server reaches a disk capacity issue due to the type of service being implemented it may be useful to delay enough to allow one test to finish before starting another.

That's why I requested being able to optionally allow the locust workers to respond differently based on the responses received with default behavior being the current behavior. Sometimes tests need to be able to respond differently.

[picklit]

I know closed, but one reason to retry is that is what would happen in the real world!