Enabled MADV_DONTNEED in systemd and docker files

[urso]

The go runtime defaults to MADV_FREE by default. In that case pages are still assigned with the Beat, even after the pages have been "returned" to the OS. The kernel will eventually claim these pages, but RSS memory usage is reported as very high (which it is not necessarily). This seems to mess with the OOM killer at times.

As workaround one can tell the runtime to use MADV_DONTNEED via GODEBUG="madvdontneed=1". We should introduce this environment variable to the systemd file, init file, and docker file by default.

What to do

• Add GODEBUG="madvdontneed=1" to docker images.
• Add GODEBUG="madvdontneed=1" to systemd unit file.
• Add GODEBUG="madvdontneed=1" to init file.
• Add a note about this in known issues for 7.x maintained versions.
• Backport the changes to 7.10 too.

More information

• madvdontneed=1 will be the default in Go 1.16 https://go-review.googlesource.com/c/go/+/267100/
• https://blog.detectify.com/2019/09/05/how-we-tracked-down-a-memory-leak-in-one-of-our-go-microservices/

[elasticmachine]

Pinging @elastic/integrations-services (Team:Services)

[fearful-symmetry]

@urso The systemd unit files are pretty straight forward, and I assume we'll just want to add an ENV to the dockerfile. However, I'm not quite sure how the init shell scripts work. Can we just... export something here?

exec /usr/share/{{.BeatName}}/bin/{{.BeatName}} \
  --path.home /usr/share/{{.BeatName}} \
  --path.config /etc/{{.BeatName}} \
  --path.data /var/lib/{{.BeatName}} \
  --path.logs /var/log/{{.BeatName}} \
  "$@"

[urso]

I assume we'll just want to add an ENV to the dockerfile.

Yes

Can we just... export something here?

This looks like a beat wrapper script for CLI use. Is it beatname.sh.tmpl?

The wrapper is ignored by the init file and by the systemd unit file. Modifying it will only affect users running Beats on command line, or those having custom scripts running the wrapper for whatever reason. Yet we should apply the change to the wraper as well. In the wrapper ensure that we do not overwrite GODEBUG if it is already set.

You find the debian init files in dev-tools/packaging/templates/deb/. The ones for the RPMs in dev-tools/packaging/templates/rpm/.

The init scripts are normal shell scripts that spawn sub-processes. In each script it is just a matter of adding export GODEBUG=${GODEBUG-"madvdontneed=1"} to the script. Maybe introduce a DEFAULT_GODEBUG="..." variable and write export GODEBUG=${GODEBUG-$DEFAULT_GODEBUG}. The - ensures that we do not modify GODEBUG even if it is empty. This allows users to "remove" our flags by running GODEBUG= filebeat.sh ...

[fearful-symmetry]

Linked PR. Not sure how to test this beyond "the env var is in the build"

[fearful-symmetry]

Merged, with backported changes. Closing this now.

[urso]

@fearful-symmetry I reopened the issue. The task "Add a note about this in known issues for 7.x maintained versions." seems to be missing. Please add a "known issue" to our docs. The oldest affected version is 7.2 I think.

[urso]

@fearful-symmetry Did we finish all tasks? Can we close the issue?

[fearful-symmetry]

Yep.