Random 403 UAEX errors when updating circuit breakers configuration

[michelgz]

Describe the bug
When updating Ambassador module circuit_breakers configuration, a subset of requests (0.1%~) fail seemingly randomly with a 403 UAEX error returned by Ambassador

To Reproduce
Steps to reproduce the behavior:

1. Update any circuit_breakers configuration in your Ambassador module, e.g:

circuit_breakers:
      - max_connections: 1000000
        max_pending_requests: 100000000
        max_requests: 100000000
        max_retries: 100000000
        priority: default

2. Ambassador picks up new configuration
3. See random 403 UAEX responses, e.g:

2023-10-26T15:53:13.056908247Z stdout F ACCESS [2023-10-26T15:53:12.893Z] "POST /example HTTP/1.1" 403 UAEX 1151 0 86 - "example" "Go-http-client/1.1" "example" "example" "-"

Expected behavior
No 403 UAEX errors when updating circuit_breakers configuration, (No AuthService, nor RateLimit is enabled on this cluster)

Versions (please complete the following information):

• Ambassador: 3.7.2 (AES)
• Kubernetes environment: EKS
• Version: 1.27

Additional context

• Issue doesn't occur in AES version 2.3.1
• Issue occurs in cluster where no AuthService or RateLimit is enabled
• Issue doesn't occur when swapping the edge-stack image (3.7.2) for the emissary-ingress image (3.8.2)

Additional questions

• Is it safe to swap the edge-stack image with the emissary-ingress image?, do they have 1:1 compatibility? are there any issues that could occur when doing that?

[cindymullins-dw]

Hi @michelgz, thanks for opening an issue for this. There's several things to communicate here so I've listed them up.

1. We suggest you try Edge Stack 3.8 - this does require a license but you can get a community license which will allow you to continue using all the licensed features of Edge Stack. Edge Stack is based on Emissary so generally you should be able to switch but it's not a difference between Edge Stack and Emissary's circuit breaking causing these 403 UAEX errors - circuit breaker functionality is the same across both products.
2. You DO have to use Redis. You can use your own Redis rather than ours if you prefer, but Redis must be enabled. Edge Stack will not behave properly without it.
3. AES creates an AuthService and routes requests thru it - it can't be turned off. Even if you don’t have an authService CRD, we create one for you. You may not be actively using it but requests are being routed thru it.
4. For the 403 UAEX, please give more details on these errors like log lines or full logs. These errors are not caused directly by circuit breaker settings so it'd be good to know more about the errors for more context.
5. Typically 403 UAEX errors are caused by a Filter or Rate Limiting so please check with your team that someone else has not introduced config that is interfering here.