test-macstadium-macos10.10-x64-1 is offline because of lack of disk space

[sam-github]

https://ci.nodejs.org/computer/test-macstadium-macos10.10-x64-1/ says

Disk space is too low. Only 0.089GB left on /Users/iojs/build.

I think Jenkins took it offline automatically.

I noticed while trying to run ansible against it (in read-only mode) so I can compare a working target with the nearform targets (#1695 ), and got disk space errors, and indeed it has no space:

$ df
Filesystem    512-blocks      Used Available Capacity  iused ifree %iused  Mounted on
/dev/disk0s2   103178384 102979512         0   100% 12897296     0  100%   /
devfs                349       349         0   100%      604     0  100%   /dev
map -hosts             0         0         0   100%        0     0  100%   /net
map auto_home          0         0         0   100%        0     0  100%   /home

[sam-github]

 17G    /Users/iojs/.
8.5G    /Users/iojs/build
4.2G    /Users/iojs/.ccache
3.3G    /Users/iojs/tmp
1.1G    /Users/iojs/Library

ccache.conf:

max_size = 5.0G

That might be unreasonably high given the disk space limits. AFAICT, docker is the only platform that configures ccache limits, every other one just lets ccache write its default conf on first run.

~/tmp/ is entirely composed of 95 directories that look like:

 find .tmp.95 -type f -ls
132799081    73000 -rwxr-xr-x    1 iojs             iojs             37374188 28 May  2018 .tmp.95/install/bin/node
132799177        8 -rw-r--r--    1 iojs             iojs                   36 28 May  2018 .tmp.95/install/lib/node/foo.js

They are all from May 28.

I removed them.

The workspaces look as expected, huge, but since citgm and node-test both require a full nodejs source build, I think the space they use is required, unless citgm can do a make distclean as @rvagg suggested elsewhere?

$ du -hs *
4.4G    citgm-smoker
 37M    libuv-test-commit-osx
4.0G    node-test-commit-osx
115M    node-test-node-addon-api-new

[sam-github]

I brought the machine back online.

@nodejs/citgm Is the ~/tmp stuff current citgm behaviour? I think citgm has improved its tmp space handling, so that might be cruft that is no longer created.

I checked the other macs, ~iojs/tmp is zero size everywhere except one that has a /Users/iojs/tmp/citgm_tmp of 613M from Oct 7, so things look OK.

[richardlau]

CITGM has been redirecting os.tmpdir() since nodejs/citgm@4974854 (back in April) (and os.homedir() since nodejs/citgm@5421cf6 two days ago).

~/tmp/ is entirely composed of 95 directories that look like:

 find .tmp.95 -type f -ls
132799081    73000 -rwxr-xr-x    1 iojs             iojs             37374188 28 May  2018 .tmp.95/install/bin/node
132799177        8 -rw-r--r--    1 iojs             iojs                   36 28 May  2018 .tmp.95/install/lib/node/foo.js

This looks like artifacts from test/parallel/test-module-loading-globalpaths.js. 95 seems like a lot... perhaps it was a (manual?) stress run?

[sam-github]

I was looking for who to ping about that test, but you wrote it :-).

Does this indicate the need for all the macs, or all the machines, really, to have a cron jobs that cleans up build artifacts more than a couple days old?

WRT to the ~/tmp/ space specifically, is its cleanup handled in some other way?

It seems like we might need a meta issue to add recurring cleanup processes to all our hosts, since our jobs do cause data to be generated outside the build workspace.

[richardlau]

I was looking for who to ping about that test, but you wrote it :-).

😅

Does this indicate the need for all the macs, or all the machines, really, to have a cron jobs that cleans up build artifacts more than a couple days old?

Maybe.

WRT to the ~/tmp/ space specifically, is its cleanup handled in some other way?

For Node.js core tests on the CI the NODE_TEST_DIR environment variable is used (on test-macstadium-macos10.11-x64-1 this appears to be /Users/iojs/tmp). I think these are not cleaned up until a subsequent test calls tmpdir.refresh().

(NODE_TEST_DIR is usually outside of the workspace as otherwise we end up hitting the path limit for Unix sockets. CITGM jobs were recently modified to use the same NODE_TEST_DIR as some of the modules tested used Unix sockets and were hitting the same path limits.)

It seems like we might need a meta issue to add recurring cleanup processes to all our hosts, since our jobs do cause data to be generated outside the build workspace.

[richardlau]

On further investigation common/tmpdir does install an exit hook to clean up since nodejs/node@8ef68e6 back in July.

[sam-github]

OK, so this is side-effect of things that have been fixed. I'm sure our CI could learn to be better about cleaning up, but the issues here look to be fixed.