Temporary backup files not always cleaned up

[lognaturel]

Our QA server at https://test.central.getodk.org/ now has a ~7GB database. Its harddrive periodically fills up and I've identified that this is because of temporary backup files that stick around. It doesn't look like every backup attempt leaves temporary files behind and I haven't figured out why/when it happens.

The only way I know how to recover is to go to /var/lib/docker/overlay2, run du -a | sort -n -r | head -n 10 to identify the biggest directories, go to the tmp directory for the biggest ones and clear out anything there.

Here are some ideas for trying to troubleshoot:

• Look at date/time stamps for tmp directories and connect to backup audit and logs
• Observe periodic backup (~6pm PDT) and see if directory gets cleaned up
• See if directory gets cleaned up after adhoc backup

[issa-tseng]

hmm. i use tmp! https://www.npmjs.com/package/tmp

maybe there's been an update that fixes this?

[issa-tseng]

also there is no guarantee, afaik, that upon calling dispose from userland the tmpdata will immediately go away. i think it's up to the OS when it happens.

[lognaturel]

i think it's up to the OS when it happens.

That occurred to me as a possibility. There were 5 directories when I cleared it earlier and I /think/ it's been more than 5 days (so 5 backups) since I last did it but I'm not sure. I didn't think to do the things in the checklist before I deleted which is why I've filed the issue. I vaguely remember doing check 3 but I didn't write down any notes so I'm not sure.

[lognaturel]

A backup has successfully run since the last message and there are no leftover tmp files. However, my manual clearing of the tmp directory left the server in a bad state resulting in this Sentry error (credentials needed). It looks like the tmp library may have been remembering a subdirectory it created previously and expecting to put files there. I wouldn't expect that to cause a crash.

When I look at the tmp directory now (/var/lib/docker/overlay2/38a511a60b09ddc9c50ec91bdc6a7acba86ee8d68851a3aa46ec5a390953f7bd/merged/tmp) I see subdirectories that contain submissions I made. I'm surprised those aren't cleaned up immediately.