System cron service memory leak

[stephendrake]

RancherOS Version: v1.2.0
Where are you running RancherOS? VMs, Bare metal, cloud, everywhere it will run.

The system container cron service (system-cron) appears to have a memory / fd leak. This is more noticeable on systems with higher uptimes.

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
  653 root      20   0  383332 115088   5816 S   0.0  1.5  64:39.52 container-cront

[root@rancheros ~]# nice lsof 2>&1 | grep "^cont.*$(pgrep container-cron)" | wc -l
39240

Compare this to when the process has just started:

  PID USER      PR  NI    VIRT    RES    SHR S %CPU %MEM     TIME+ COMMAND
 590 root      20   0   36268   9536   5832 S  0.0  0.2   0:00.03 container-cront

[root@rancheros ~]# nice lsof 2>&1 | grep "^cont.*$(pgrep container-cron)" | wc -l
96

This might be solved by version v0.3.0 of the container crontab, but that version can't be used until the system-docker version has been upgraded (#2255).

[root@rancheros ~]# system-docker ps | grep test-cron
147806b99cc0        rancher/container-crontab:v0.3.0                                       "container-crontab"      6 minutes ago       Restarting (1) 2 minutes ago                       test-cron
[root@rancheros ~]# system-docker logs -f test-cron
time="2018-02-20T09:25:18Z" level=info msg="Starting Cron" 
time="2018-02-20T09:25:18Z" level=fatal msg="Error response from daemon: client is newer than server (client API version: 1.26, server API version: 1.23)" 

[lost-carrier]

I think I see the the memory issue as well... 770MB used by the container-crontab after about 5 days of uptime...

system-docker restart system-cron freed that memory...

[kingsd041]

The system-docker container-crontab of rancheros v1.3.0-rc1 has been upgraded to v0.4.0 and the issue has been fixed.