flake on IpVersions/AdmissionControlIntegrationTest.GrpcTest/1 in admission_control_integration_test

[alyssawilk]

@tony612 could you take a look?

https://dev.azure.com/cncf/4684fb3d-0389-4e0b-8251-221942316e06/_apis/build/builds/79778/logs/271

2021-06-21T15:29:48.9188031Z external/envoy/test/extensions/filters/http/admission_control/admission_control_integration_test.cc:162: Failure
2021-06-21T15:29:48.9188947Z The difference between throttle_count / request_count and 0.80 is 0.056666666666666643, which exceeds 0.05, where
2021-06-21T15:29:48.9189439Z throttle_count / request_count evaluates to 0.85666666666666669,
2021-06-21T15:29:48.9189843Z 0.80 evaluates to 0.80000000000000004, and
2021-06-21T15:29:48.9190213Z 0.05 evaluates to 0.050000000000000003.
2021-06-21T15:29:48.9190540Z Stack trace:
2021-06-21T15:29:48.9191311Z 0x216f481: Envoy::(anonymous namespace)::AdmissionControlIntegrationTest_GrpcTest_Test::TestBody()
2021-06-21T15:29:48.9191845Z 0x4d564d4: testing::internal::HandleSehExceptionsInMethodIfSupported<>()
2021-06-21T15:29:48.9192319Z 0x4d4687e: testing::internal::HandleExceptionsInMethodIfSupported<>()
2021-06-21T15:29:48.9192757Z 0x4d2eec3: testing::Test::Run()
2021-06-21T15:29:48.9193304Z 0x4d2fabd: testing::TestInfo::Run()
2021-06-21T15:29:48.9193673Z ... Google Test internal frames ...
2021-06-21T15:29:48.9193870Z
2021-06-21T15:29:48.9194740Z [ FAILED ] IpVersions/AdmissionControlIntegrationTest.GrpcTest/1, where GetParam() = 4-byte object <01-00 00-00> (9220 ms)
2021-06-21T15:29:48.9195516Z [----------] 4 tests from IpVersions/AdmissionControlIntegrationTest (36826 ms total)
2021-06-21T15:29:48.9195793Z
2021-06-21T15:29:48.9196273Z [----------] Global test environment tear-down
2021-06-21T15:29:48.9197027Z [==========] 4 tests from 1 test suite ran. (36826 ms total)
2021-06-21T15:29:48.9197404Z [ PASSED ] 3 tests.
2021-06-21T15:29:48.9197751Z [ FAILED ] 1 test, listed below:
2021-06-21T15:29:48.9198458Z [ FAILED ] IpVersions/AdmissionControlIntegrationTest.GrpcTest/1, where GetParam() = 4-byte object <01-00 00-00>

[tonya11en]

Looking into it this afternoon.

[tonya11en]

So I think what's going on here is that #16742 added the default upper-bound of 80% on rejection probability, which results in a slower convergence of the observed rejection rate to what we expect. I hacked up a script to see the variability of the error for 300 requests across 1000 tests:

As the upper-bound drops, the variance in observed errors increases. I expect there to have been around ~20% flakiness in those tests since the patch was merged. Increasing the allowed error to 10% as you did in #17084 would still probably yield some flakiness...

I think the best option here is to just increase the rejection upper-bound to 100% in the test and leave the allowed absolute error at 10% to put this thing to rest. Sorry for the flakiness this thing has caused over the last year or so.

I'll follow-up with a patch.

[alyssawilk]

Awesome. Thanks so much for looking into this!