Add event batch processing results and rerun reconfig test

[bjee19]

Proposed changes

Add event batch processing metrics to reconfiguration test and rerun the test. Added some clarification and other small adjustments to test.

Problem: We already measured how long it took for NGINX to reload, but didn't measure for how long it took for NGF to update statuses of a resource.

Solution: Add event batch processing metrics to test.

Testing: Reran the manual test.

Closes #1127

Checklist

Before creating a PR, run through this checklist and mark each as complete.

• I have read the CONTRIBUTING doc
• I have added tests that prove my fix is effective or that my feature works
• I have checked that all unit tests pass after adding my changes
• I have updated necessary documentation
• I have rebased my branch onto main
• I will ensure my PR is targeting the main branch and pulling from my branch from my own fork

[bjee19]

Note the quite big changes in results of the TimeToReadyTotal and TimeToReadyAvgSingle, I checked with @ciarams87 with a couple of them and she said that they seemed reasonable. What do you all think?

[ciarams87]

Note the quite big changes in results of the TimeToReadyTotal and TimeToReadyAvgSingle, I checked with @ciarams87 with a couple of them and she said that they seemed reasonable. What do you all think?

The big deviation is mostly in test 2. I believe this expected behaviour after the status updater improvements (only updating the status when there is a change). My understanding is that there are actually less changes being processed, which then leads to more efficient handling of each individual resource update. This means we are doing less batching (that is, there are less items being added to each batch), and therefore calling a reload for almost every config change. Because we are reloading more, I think this is leading to the increased TTR.

Unfortunately we don't have the event batch processing results from the first round, but I checked the logs from the run and there are less batches with more resources in each (cumulatively increasing with each batch) and correspondingly less reloads. The largest batch resulting in a config change in the old runs logs was 310.

Does that make sense?

[kate-osborn]

🚀