Possible performance regression when upgrading to 0.15.2

[timflannagan]

We recently bumped our controller runtime dependency to the 0.15.2 release. After running our internal performance suite against a build with the 0.15.2 c-r version, we noticed roughly a ~10-15% performance regression compared to previous builds that used the 0.15.0 c-r version. When gathering and analyzing pprof profiles from those affected builds, our controller was spending a lot more time in the MatchingLabels ApplyToList client method for that type compared to 0.15.0 builds.

Looking at the changelog diff between 0.15.0 and 0.15.2, we can make the assumption that the culprit for this regression at scale was #2098 due to the changes made to the client package. For additional context, our controller extensively uses the client.MatchingLabels list option when making client list calls.

[troy0820]

/kind bug support

[alvaroaleman]

The previous behavior was buggy. If you have ideas on how to speed up the non-buggy implementation, feel free to suggest them or raise a PR.

[timflannagan]

@alvaroaleman Thanks for the quick reply - I'll see if I can shore up some cycles to look into this further. I'm guessing the issue is we're constantly re-shuffling the underlying set when adding new label selector requirements.