Reduce resource consumption of test compilation by simplifying mock library inclusions

[sunjayBhatia]

Description:
Many tests/test libraries over-include especially mock libraries which causes increased memory usage, disk usage, and compilation time. This is especially a problem on Windows with cl.exe but we have also observed significant resource usage that can be reduced by as much as half on Linux with clang.

Attached to this issue is a file showing the peak working set memory usage of cl.exe compiling various tests/test libraries (units in KB). We can see many of the tests cause the compiler to use multiple GB of memory, often for simple tests that do not actually require many of the includes they are being built with. We have been able to reduce this usage (see linked PR) by cleaning up these includes. Our primary focus has been enabling faster/less resource intensive compilation of tests in Windows CI but this endeavor seems like it would be relevant to compilation performance on other platforms. We can potentially reduce compilation time, disk usage of artifacts/parameter files, and memory usage of the compiler (by orders of magnitude in some cases, multiple GB reduction in others).

While we have had some success empirically assessing the memory usage of compiling tests and finding the most expensive, it is not super feasible to go one-by-one and manually assess/fix the includes etc. to pare-down each test. We are hoping to potentially get some help reducing the overhead of compiling these tests from other contributors working toward similar changes as in the PR linked below as well as finding a programmatic solution that could potentially be automated (in CI or otherwise) that could identify over-inclusion of mocks and other large libraries that are not used. Other strategies to help solve this could be further breaking down larger libraries with many classes into their component parts so tests only include exactly what interfaces etc. they need.

Relevant Links:

• See Lower heap and disk space used by compiling kafka tests #10915 for a relevant PR that addresses this for the kafka network filter extension tests
• Gist with peak working set memory usage for tests that exceed 2GB: https://gist.github.com/sunjayBhatia/d2d4a0af1eab16777f94a4dc4959b342
• Powershell script used to collect data: https://github.com/greenhouse-org/envoy/blob/84fca545722432a48371fca1ced02253cddb86d0/get-cl-workingset-peak.ps1

[sunjayBhatia]

cc @wrowe

[sunjayBhatia]

cc @mattklein123 @lizan as you have been working on improving build performance in CI etc.

[mattklein123]

See #8770. cc also @htuch as we have noted understanding and improving compile performance as a possible intern project for this summer. I would love to make progress on this.

[htuch]

I think this is also related to getting https://include-what-you-use.org/ working for Envoy. Last time I looked it needed a custom toolchain but I think this has changed.

[ahedberg]

/assign ahedberg

[ahedberg]

/assign foreseeable

[repokitteh-read-only]

foreseeable cannot be assigned to this issue.

🐱

Caused by: a #10917 (comment) was created by @ahedberg.

see: more, trace.

[ahedberg]

/assign foreseeable

[asraa]

Nice!! I just saw this. I have been thinking about this in the context of our frequent timeouts and OOMs in OSS-Fuzz. I ran a few bazel performance analysis on some fuzz targets and found bottlenecks in server/mocks.cc and in gperftools:

   39.021 s   42.47%   action 'CcConfigureMakeRule external/envoy/bazel/foreign_cc/gperftools_build/include'
   40.090 s   43.63%   action 'Compiling test/mocks/server/mocks.cc'

I had some draft patches splitting up server/mocks.cc as well, and I'm happy that this affects others, as I will happily help review/breakdown mock libraries.

[foreseeable]

I used Bazel to generated a few performance profiles for some test after simplifying some mock library inclusions:
https://drive.google.com/file/d/1O8B0mLy-VjNXLRSqquNHYaa7izp4Sk7b/view?usp=sharing

(one can open them from chrome://tracing with chrome)

The performance bottlenecks for those tests before was building the monolithic mock headers. But now I can not figure out what to optimize for next step. Can someone have a look of them? Appreciated.

[lizan]

@foreseeable Great job. I think we can call this issue done.

For the whole build performance, it would be nice if you can take a look why compiling test.cc take that long time. If you pass -ftime-trace to the compiler (via bazel --copt or --per_file_copt), you should be able to get details of that.

[sunjayBhatia]

@foreseeable Great job. I think we can call this issue done.

Closing this as per this comment, if this needs to be reopened, I can do so