Add runtime stats logger and pprof

[angelcar]

Summary

This change adds a new log file: runtime-stats.log, which will contain the go runtime statistics. The intention is to have more information at runtime regarding, memory, CPU, and goroutine usage.

In addition, the ECS_ENABLE_PPROF configuration is introduced in order to enable pprof.

Both of these changes aim to better assess issues such as #2865.

Implementation details

Runtime stats are retrieved by periodically executing runtime.ReadMemStats, and runtime.NumGoroutine. The results of these methods are then logged to the new runtime-stats.log file. Stats are logged every 5 minutes and the interval is not configurable since there seems to be no justification for it. Also it is not advisable to keep increasing the list of configuration variables if not strictly necessary as the list is already large and might confusing to some.
This log is going to be enabled and considered to be just another log file part of the agent.

Pprof is only enabled on demand via the ECS_ENABLE_PPROF configuration. In this case, a new configuration was needed since users might not want/need to have pprof enabled in production environments; hence, the configuration is disabled by default.

Testing

Manually rested that pprof is enabled and functional when ECS_ENABLE_PPROF=true, and not reachable otherwise.

Also tested that the runtime stats log logs successfully every 5 minutes. Below is a sample log entry:

FreesCount=416081, HeapMemoryAlloc=5881688, HeapMemoryInUse=8232960, HeapMemoryIdle=57434112, HeapMemoryReleased=56737792, MallocCount=441929, StackMemoryInUse=1441792, GarbageCollectionCount=7, GarbageCollectionTime=453.039µs, GCCPUFraction=0.000016, GoroutineCount=124, UpTime=3h25m0.000145476s

UTs were also added/modified as necessary/

New tests cover the changes: yes

Description for the changelog

• Add runtime-stats log file to periodically log agent's runtime stats such as used memory and CPU
• Add new configuration setting to enable/disable pprof

Licensing

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

[fenxiong]

looks good one question

[sharanyad]

is there any reason why pprof output is through introspection endpoint, while runtime stats is through a file?
why not redirect pprof output to a file, so we can capture those as part of ECS logs collector as well?

[angelcar]

is there any reason why pprof output is through introspection endpoint, while runtime stats is through a file?
why not redirect pprof output to a file, so we can capture those as part of ECS logs collector as well?

The http pprof endpoints are a standard way in which go exposes profiling data. For example, tools like go tool pprof can communicate with these endpoints to retrieve and interpret profiling information. If a file is needed at some point, it can be easily generated by invoking the pprof http endpoint with curl and redirecting the output to a file (like I documented on the readme file of this PR). For more information see: https://pkg.go.dev/net/http/pprof