Set Go garbage collection target percentage to 400%

After removal of memory_ballast extension in v0.97.0, the Go garbage collection is running more aggressively, which
increased CPU usage and leads to reduced throughput of the collector. This change reduces the frequency of garbage
collection cycles to improves performance of the collector for typical workloads. As a result, the collector will
report higher memory usage, but it will be bound to the same configured limits. If you want to revert to the previous
behavior, set the `GOGC` environment variable to `100`.

CHANGELOG.md

@@ -2,6 +2,15 @@
 
 ## Unreleased
 
+### 💡 Enhancements 💡
+
+- (Splunk) Set Go garbage collection target percentage to 400% ([#5034](https://github.com/signalfx/splunk-otel-collector/pull/5034))
+  After removal of memory_ballast extension in v0.97.0, the Go garbage collection is running more aggressively, which 
+  increased CPU usage and leads to reduced throughput of the collector. This change reduces the frequency of garbage 
+  collection cycles to improves performance of the collector for typical workloads. As a result, the collector will
+  report higher memory usage, but it will be bound to the same configured limits. If you want to revert to the previous
+  behavior, set the `GOGC` environment variable to `100`.
+
 ### 🧰 Bug fixes 🧰
 
 - (Splunk) `receiver/discovery`: Do not emit entity events for discovered endpoints that are not evaluated yet

internal/settings/settings.go

@@ -43,6 +43,7 @@ const (
 	HecLogIngestURLEnvVar     = "SPLUNK_HEC_URL"
 	ListenInterfaceEnvVar     = "SPLUNK_LISTEN_INTERFACE"
 	GoMemLimitEnvVar          = "GOMEMLIMIT"
+	GoGCEnvVar                = "GOGC"
 	// nolint:gosec
 	HecTokenEnvVar    = "SPLUNK_HEC_TOKEN" // this isn't a hardcoded token
 	IngestURLEnvVar   = "SPLUNK_INGEST_URL"
@@ -59,6 +60,7 @@ const (
 
 	DefaultMemoryLimitPercentage = 90
 	DefaultMemoryTotalMiB        = 512
+	DefaultGoGC                  = 400
 	DefaultListenInterface       = "0.0.0.0"
 	DefaultAgentConfigLinux      = "/etc/otel/collector/agent_config.yaml"
 	featureGates                 = "feature-gates"
@@ -371,6 +373,11 @@ func checkRuntimeParams(settings *Settings) error {
 		setSoftMemoryLimit(memTotalSize)
 	}
 
+	if _, ok := os.LookupEnv(GoGCEnvVar); !ok {
+		debug.SetGCPercent(DefaultGoGC)
+		log.Printf("GOGC is set to %d by default", DefaultGoGC)
+	}
+
 	return nil
 }
 
@@ -517,6 +524,14 @@ func setSoftMemoryLimit(memTotalSizeMiB int) {
 	log.Printf("Set soft memory limit set to %d MiB", memLimit)
 }
 
+// Check if the GOMEMLIMIT is specified via the env var, if not set the soft memory limit to 90% of the MemLimitMiBEnvVar
+func setSoftMemoryLimit(memTotalSizeMiB int) {
+	memLimit := int64(memTotalSizeMiB * DefaultMemoryLimitPercentage / 100)
+	// 1 MiB = 1048576 bytes
+	debug.SetMemoryLimit(memLimit * 1048576)
+	log.Printf("Set soft memory limit set to %d MiB", memLimit)
+}
+
 // Validate and set the memory limit
 func setMemoryLimit(memTotalSizeMiB int) (int, error) {
 	memLimit := memTotalSizeMiB * DefaultMemoryLimitPercentage / 100