runtime: Memory not returned to system on linux

[TimmyOVO]

What version of Go are you using (go version)?

$ go version
go version go1.13.3 linux/amd64

Does this issue reproduce with the latest release?

yes

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/root/.cache/go-build"
GOENV="/root/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/root/go"
GOPRIVATE=""
GOPROXY="direct"
GOROOT="/usr/lib/golang"
GOSUMDB="off"
GOTMPDIR=""
GOTOOLDIR="/usr/lib/golang/pkg/tool/linux_amd64"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build842407928=/tmp/go-build -gno-record-gcc-switches"

What did you do?

import (
	"runtime/debug"
	"time"
)

func main() {
	bytes := make([]byte, 326766666)
	for k, _ := range bytes {
		bytes[k] = 0
	}
	bytes = nil
	go func() {
		time.Sleep(time.Minute * 5)
		println("after 5 minutes")
	}()
	time.Sleep(time.Minute * 20)
}

What did you expect to see?

memory should be returned to system

What did you see instead?

On macos the memory be returned to system after about 3 minutes, but after waiting for 20 minutes, the memory still being use by the application on linux

[randall77]

Does using runtime/debug.FreeOSMemory help?
Does Go 1.14 help?

What method are you using to measure RSS? A common confusion is that while Go releases memory back to the OS, if the OS has no other demand for that memory it won't actually take the pages back. Only if there are other demands for that memory will the Go process's RSS go down.

Please don't post screenshots of code or logs. It makes it hard for us to reproduce your results. Just copy-paste the text directly.

[TimmyOVO]

update Golang to 1.14 doesn't help
using debug.FreeOSMemory either
but after downgrade Golang to 1.11 works

the code I'm using for test

package main

import (
	"runtime/debug"
	"time"
)

func main() {
	bytes := make([]byte, 326766666)
	for k, _ := range bytes {
		bytes[k] = 0
	}
	bytes = nil
	go func() {
		time.Sleep(time.Minute * 5)
		println("after 5 minutes")
		debug.FreeOSMemory()
	}()
	time.Sleep(time.Minute * 20)
}

Compiled by Golang 1.11.13

go version go1.11.13 darwin/amd64

[thepudds]

@TimmyOVO Are you on a relatively recent Linux?

If so, what does this show for the PID of your process:

grep LazyFree /proc/<PID>/smaps | grep -v " 0 kB"

From /proc man page:

LazyFree %lu (since Linux 4.12)
          Shows the amount of memory marked by madvise(2) MADV_FREE.

From Go 1.12 release notes:

On Linux, the runtime now uses MADV_FREE to release unused memory. This is more efficient but may result in higher reported RSS. The kernel will reclaim the unused data when it is needed.

(On mobile, so brief).

[thepudds]

I updated @TimmyOVO's code from above (#37585 (comment)) to periodically output RSS memory values as reported by ps and top, as well as to output the sum of LazyFree values from /proc/<pid>/smaps.

Full update is on playground, but the primary addition is this quick & dirty function:

reportMem := func(header string) {
	log.Println("---", header, "---")
	pid := os.Getpid()
	shell("ps rss:  ", `ps -o rss= -p %d`, pid)
	shell("top res: ", `top -b -n 1 | awk '/^[ ]*%d[ ]/ {print $6}'`, pid)
	shell("LazyFree:", `cat /proc/%d/smaps | awk '/LazyFree/ {sum+= $2} END {print sum}'`, pid)
}

I think the results are as expected for Go 1.11, 1.13, and 1.14, at least as far as I understand things in this general area:

• In 1.11, RSS declines significantly after FreeOSMemory.
• In 1.13 and 1.14, after 5 minutes RSS is still high while LazyFree has climbed to almost the same value as RSS (>300MB), and the subsequent FreeOSMemory does not seem to have material impact.

Go 1.11

$ go1.11.10 run main.go 
2020/02/29 22:33:01 --- before alloc ---
2020/02/29 22:33:01 ps rss:   1544
2020/02/29 22:33:02 top res:  1544
2020/02/29 22:33:02 LazyFree: 0
2020/02/29 22:33:02 --- after alloc ---
2020/02/29 22:33:02 ps rss:   331076
2020/02/29 22:33:02 top res:  331076
2020/02/29 22:33:02 LazyFree: 0
2020/02/29 22:38:02 --- after 5 minutes --- 
2020/02/29 22:38:02 ps rss:   331076
2020/02/29 22:38:02 top res:  331076
2020/02/29 22:38:02 LazyFree: 0
2020/02/29 22:38:02 --- after FreeOSMemory ---
2020/02/29 22:38:02 ps rss:   11908
2020/02/29 22:38:03 top res:  11908
2020/02/29 22:38:03 LazyFree: 0

Go 1.13

go1.13.3 run main.go
2020/02/29 22:40:05 --- before alloc ---
2020/02/29 22:40:05 ps rss:   1572
2020/02/29 22:40:06 top res:  1572
2020/02/29 22:40:06 LazyFree: 0
2020/02/29 22:40:06 --- after alloc ---
2020/02/29 22:40:06 ps rss:   331320
2020/02/29 22:40:06 top res:  331320
2020/02/29 22:40:06 LazyFree: 2384
2020/02/29 22:45:06 --- after 5 minutes ---
2020/02/29 22:45:06 ps rss:   331320
2020/02/29 22:45:06 top res:  331320
2020/02/29 22:45:06 LazyFree: 319104
2020/02/29 22:45:06 --- after FreeOSMemory ---
2020/02/29 22:45:06 ps rss:   331320
2020/02/29 22:45:06 top res:  331320
2020/02/29 22:45:06 LazyFree: 319032

Go 1.14

$ go1.14 run main.go 
2020/02/29 22:47:13 --- before alloc ---
2020/02/29 22:47:13 ps rss:   1852
2020/02/29 22:47:14 top res:  1852
2020/02/29 22:47:14 LazyFree: 0
2020/02/29 22:47:14 --- after alloc ---
2020/02/29 22:47:14 ps rss:   331312
2020/02/29 22:47:14 top res:  331312
2020/02/29 22:47:14 LazyFree: 0
2020/02/29 22:52:14 --- after 5 minutes ---
2020/02/29 22:52:14 ps rss:   331312
2020/02/29 22:52:14 top res:  331312
2020/02/29 22:52:14 LazyFree: 319352
2020/02/29 22:52:14 --- after FreeOSMemory ---
2020/02/29 22:52:14 ps rss:   331576
2020/02/29 22:52:14 top res:  331576
2020/02/29 22:52:14 LazyFree: 319372

This is on Ubuntu:

$ grep DESC /etc/lsb-release
DISTRIB_DESCRIPTION="Ubuntu 18.04.2 LTS"
$ uname -a
Linux instance-fzgo-test-7 4.15.0-1037-gcp #39-Ubuntu SMP Wed Jul 3 06:28:59 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

[bonedaddy]

If you run your binary with GODEBUG=madvdontneed=1 memory will be returned to the OS

[gopherbot]

Change https://golang.org/cl/267100 mentions this issue: runtime: default to MADV_DONTNEED on Linux

[changkun]

Because 05e6d28 is landed on the master, this issue can be closed.

[odeke-em]

Thank you @changkun for the notice, and @aclements for the fix, also happy New Year everyone!