Filebeat Httpjson input needs to handle Rate-limit error gracefully

[alakahakai]

Filebeat Httpjson input needs to handle Rate-limit error (Http Error code 429) gracefully when the proposed time delay by Okta ends up not sufficient.

[elasticmachine]

Pinging @elastic/siem (Team:SIEM)

[andrewkroh]

So it seems like with the org wide rate limit, that the rate limit headers are not fully reliable since all applications are sharing from the same token bucket. So even if Filebeat honors the X-Rate-Limit-Reset, other concurrently running applications might have already used up the available requests. So Filebeat will have to keep trying.

Additionally, the X-Rate-Limit-Reset time for concurrent rate limits is only a suggestion. There's no guarantee that enough requests will complete to stop exceeding the concurrent rate limit at the time indicated.

Source: https://developer.okta.com/docs/reference/rate-limits/#check-your-rate-limits-with-okta-s-rate-limit-headers