Memory leak, the memory occupied by waiting several hours will not be returned to the operating system

[franklwel1990]

What version of Go are you using (go version)?

go version go1.15.3 linux/amd64

Does this issue reproduce with the latest release?

yes

What operating system and processor architecture are you using (go env)?

go env Output

GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/root/.cache/go-build"
GOENV="/root/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/jhw/www/html/gopath/pkg/mod"
GONOPROXY="gitlab.com/idmabar,bitbucket.org/idmabar,github.com/idmabar"
GONOSUMDB="gitlab.com/idmabar,bitbucket.org/idmabar,github.com/idmabar"
GOOS="linux"
GOPATH="/jhw/www/html/gopath"
GOPRIVATE="gitlab.com/idmabar,bitbucket.org/idmabar,github.com/idmabar"
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/jhw/www/html/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/jhw/www/html/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build434819960=/tmp/go-build -gno-record-gcc-switches"

What did you do?

通过go-xorm/xorm读取500万行数据，占用linux的内存used一直不会被释放

code:
d := make([]TpccTestingDetail, 0, 5000000)
err := app.DB.Table(req.Table).Select("s_i_id,s_w_id,s_quantity,s_dist_01,s_dist_02,s_dist_03,s_dist_04,s_dist_05,s_dist_06,s_dist_07,s_dist_08,s_dist_09,s_dist_10,s_ytd,s_order_cnt,s_remote_cnt,s_data").Limit(5000000, 0).Find(&d)

sql查询后（大小是kb）：
Alloc = 5533008 TotalAlloc = 3285136 Sys = 8489336 HeapReleased = 52560 HeapAlloc = 5533008 HeapSys = 7994656 HeapIdle = 1383728 HeapInuse = 6610928 NumGC = 15

手动runtime.GC()后（大小是kb）：
Alloc = 983 TotalAlloc = 3285136 Sys = 8489336 HeapReleased = 52528 HeapAlloc = 983 HeapSys = 7994656 HeapIdle = 7992224 HeapInuse = 2432 NumGC = 16

手动debug.FreeOSMemory()后（大小是kb）：
Alloc = 984 TotalAlloc = 32851366 Sys = 8489336 HeapReleased = 7992088 HeapAlloc = 984 HeapSys = 7994656 HeapIdle = 7992240 HeapInuse = 2416 NumGC = 17

go进程(ps -p 2446):
PID TTY STAT TIME MAJFL TRS DRS RSS %MEM COMMAND
2446 pts/3 Sl+ 0:01 4 5397 2044214 32752 0.1 go-test

系统free -m 后：
total used free shared buff/cache available
Mem: 16013 13501 309 35 2202 9903
Swap: 0 0 0

What did you expect to see?

能正常释放应用程序占用的内存归还操作系统。（函数返回，操作完空闲一段时间后，手动释放）

What did you see instead?

持续观察linux的used一直没有归还操作系统，直到进程被kill掉
linux使用的内存 used 观察几个小时后都没有变化。

[fanlv]

go 1.12 版本以后申请内存是用的MADV_FREE，只有操作系统内存不够的时候（使用docker的话可能会导致这个判断不准确），才会回收。具体可以自己google下。 你可以设置 GODEBUG=madvdontneed=1 取消这个优化，在测试下。

[martisch]

See #42330. Linux also needs to request the memory back e.g. under memory pressure before stats change.

[mvdan]

What @martisch said. Also, see https://golang.org/wiki/Questions.