No space left on (CI?) device

[gireeshpunathil]

11:49:34 ERROR: [description-setter] Error while parsing logs for description-setter
11:49:34 java.io.IOException: No space left on device
11:49:34 	at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
11:49:34 	at sun.nio.ch.FileDispatcherImpl.write(FileDispatcherImpl.java:60)
11:49:34 	at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
11:49:34 	at sun.nio.ch.IOUtil.write(IOUtil.java:65)
11:49:34 	at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:211)
11:49:34 	at hudson.util.FileChannelWriter.write(FileChannelWriter.java:72)
11:49:34 	at java.io.Writer.write(Writer.java:192)

ref:
https://ci.nodejs.org/job/node-test-pull-request/26948/console
many other jobs just got truncated in between such as:
https://ci.nodejs.org/job/node-test-pull-request/26946/

which I am attributing to the same problem.

[targos]

It's also no longer possible to start jobs.

[targos]

@nodejs/build-infra

[rvagg]

on it, ci server out of space, trying to figure out where exactly

[rvagg]

Something's up with our backup server, it usually does the cleanup of our ci server but it hasn't been running. Chasing that up. In the meantime I've done a manual build removal, not as aggressive as the normal cleanup does and this may need to be repeated if we don't get the backup server sorted out soon.

For the record (probably myself but @nodejs/build-infra could do this), this is what I'm doing:

cd /var/lib/jenkins/jobs/
for dir in $(find . -name builds -type d); do find $dir -type d -regex '.*/[0-9]+' -regextype posix-extended -mtime +20 -exec rm -rf '{}' \;; done

Jenkins needs a reload/restart after that I think.

Will leave this open until we're back to auto-cleanup.

[rvagg]

New backup server is online and working, pruning old builds so we only keep the last 7 days worth.

[gireeshpunathil]

thanks Rod!