Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

No space left on device on test-linuxonecc-rhel72-s390x-2 #1757

Closed
richardlau opened this issue Apr 11, 2019 · 9 comments
Closed

No space left on device on test-linuxonecc-rhel72-s390x-2 #1757

richardlau opened this issue Apr 11, 2019 · 9 comments

Comments

@richardlau
Copy link
Member

e.g. https://ci.nodejs.org/view/Node.js-citgm/job/citgm-smoker/1820/nodes=rhel72-s390x/console,
https://ci.nodejs.org/job/libuv-test-commit-linux/nodes=rhel72-s390x/1394/console

FATAL: Unable to produce a script file
Also:   hudson.remoting.Channel$CallSiteStackTrace: Remote call to JNLP4-connect connection from 148.100.110.64/148.100.110.64:33690
		at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1741)
		at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357)
		at hudson.remoting.Channel.call(Channel.java:955)
		at hudson.FilePath.act(FilePath.java:1072)
		at hudson.FilePath.act(FilePath.java:1061)
		at hudson.FilePath.createTextTempFile(FilePath.java:1466)
		at hudson.tasks.CommandInterpreter.createScriptFile(CommandInterpreter.java:162)
		at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:94)
		at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:66)
		at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
		at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:744)
		at hudson.model.Build$BuildExecution.build(Build.java:206)
		at hudson.model.Build$BuildExecution.doRun(Build.java:163)
		at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:504)
		at hudson.model.Run.execute(Run.java:1810)
		at hudson.matrix.MatrixRun.run(MatrixRun.java:153)
		at hudson.model.ResourceController.execute(ResourceController.java:97)
		at hudson.model.Executor.run(Executor.java:429)
java.io.IOException: No space left on device
	at java.io.FileOutputStream.writeBytes(Native Method)
	at java.io.FileOutputStream.write(FileOutputStream.java:326)
	at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221)
	at sun.nio.cs.StreamEncoder.implClose(StreamEncoder.java:316)
	at sun.nio.cs.StreamEncoder.close(StreamEncoder.java:149)
	at java.io.OutputStreamWriter.close(OutputStreamWriter.java:233)
	at hudson.FilePath$CreateTextTempFile.invoke(FilePath.java:1499)
	at hudson.FilePath$CreateTextTempFile.invoke(FilePath.java:1471)
	at hudson.FilePath$FileCallableWrapper.call(FilePath.java:3086)
	at hudson.remoting.UserRequest.perform(UserRequest.java:212)
	at hudson.remoting.UserRequest.perform(UserRequest.java:54)
	at hudson.remoting.Request$2.run(Request.java:369)
	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:93)
	at java.lang.Thread.run(Thread.java:748)
Caused: java.io.IOException: Failed to create a temp file on /data/iojs/build/workspace/libuv-test-commit-linux/nodes/rhel72-s390x
	at hudson.FilePath.createTextTempFile(FilePath.java:1468)
	at hudson.tasks.CommandInterpreter.createScriptFile(CommandInterpreter.java:162)
	at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:94)
	at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:66)
	at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
	at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:744)
	at hudson.model.Build$BuildExecution.build(Build.java:206)
	at hudson.model.Build$BuildExecution.doRun(Build.java:163)
	at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:504)
	at hudson.model.Run.execute(Run.java:1810)
	at hudson.matrix.MatrixRun.run(MatrixRun.java:153)
	at hudson.model.ResourceController.execute(ResourceController.java:97)
	at hudson.model.Executor.run(Executor.java:429)
Build step 'Execute shell' marked build as failure
@sam-github
Copy link
Contributor

@mhdawson or @nodejs/build there are 1.2 Gig of foo-* files in /tmp on this host, most look like:

 31618    4 drwxr-xr-x   3 iojs     iojs         4096 Apr  5 11:36 ./foo-lpSqGy/.clinic
 31615  488 -rw-r--r--   1 iojs     iojs       496657 Apr  5 11:36 ./foo-lpSqGy/.clinic/42913.clinic-flame.html
 31619    4 drwxr-xr-x   2 iojs     iojs         4096 Apr  5 11:36 ./foo-lpSqGy/.clinic/42913.clinic-flame
 31625    4 -rw-r--r--   1 iojs     iojs          425 Apr  5 11:36 ./foo-lpSqGy/.clinic/42913.clinic-flame/42913.clinic-flame-systeminfo
 31626    4 -rw-r--r--   1 iojs     iojs            2 Apr  5 11:36 ./foo-lpSqGy/.clinic/42913.clinic-flame/42913.clinic-flame-inlinedfunctions
 31623    4 -rw-r--r--   1 iojs     iojs            2 Apr  5 11:36 ./foo-lpSqGy/.clinic/42913.clinic-flame/42913.clinic-flame-samples

what are they?

@sam-github
Copy link
Contributor

They look to be some kind of nearform flame graphs. There aren't any of these on -1 or -3.

I wonder if citgm was running node-clinic tests on this host?

@sam-github
Copy link
Contributor

https://ci.nodejs.org/job/citgm-smoker/nodes=rhel72-s390x/ smoking gun?

@sam-github
Copy link
Contributor

Home and /tmp have the most data variance between the hosts.

I wonder if citgm could set TMPDIR before running, and clean it up after?

[1] 10:08:17 [SUCCESS] test-linuxonecc-rhel72-s390x-2
328M    /tmp
2.7G    /usr
754M    /var
385M    /run
124K    /home
19G     /data
[2] 10:08:18 [SUCCESS] test-linuxonecc-rhel72-s390x-1
7.2M    /tmp
2.7G    /usr
755M    /var
409M    /run
1019M   /home
18G     /data
[3] 10:08:27 [SUCCESS] test-linuxonecc-rhel72-s390x-3
8.6M    /tmp
2.7G    /usr
1.1G    /var
57M     /run
67M     /home
19G     /data

@richardlau
Copy link
Member Author

I wonder if citgm could set TMPDIR before running, and clean it up after?

Yeah, we could either modify CITGM to spawn the tests for each module with TMPDIR set or set it in the various CITGM jobs (or both).

citgm itself currently executes with its tmpdir set to within the workspace but that just (at the moment) controls where the module is installed into and then tested from (so if the test itself is writing to TMPDIR it's currently not affected).

11:32:06 ++ citgm-all -J --nodedir=/data/iojs/build/workspace/citgm-smoker/nodes/rhel72-s390x/node -v warn -x /data/iojs/build/workspace/citgm-smoker/nodes/rhel72-s390x/smoker/report.xml -q error --tmpDir /data/iojs/build/workspace/citgm-smoker/nodes/rhel72-s390x/citgm_tmp

@mhdawson
Copy link
Member

I think having CITGM set TMPDIR and having it be within the workspace along with some cleanup would make sense.

@mhdawson
Copy link
Member

@sam-github did you clean up the drive?

@richardlau
Copy link
Member Author

richardlau commented Apr 12, 2019 •
edited
Loading

They look to be some kind of nearform flame graphs. There aren't any of these on -1 or -3.

I wonder if citgm was running node-clinic tests on this host?

Did some digging and this is a recent thing. The newest version of clinic switched from tmp to tempy for its tests (clinicjs/node-clinic#150) which meant that it no longer cleans up the temporary generated test files.

I'll submit a PR to CITGM to redirect TMPDIR (and TEMP and TMP) to a directory which CITGM will clear afterwards, which will deal with modules whose tests use os.tmpdir() (https://github.com/nodejs/node/blob/7b854959e79a1ae4cb325123ba118b83a619aaea/lib/os.js#L124-L139).

Edit: submitted nodejs/citgm#706.

@sam-github sam-github mentioned this issue Apr 12, 2019
Merged
richardlau added a commit to richardlau/citgm that referenced this issue Apr 12, 2019
@richardlau
lib: redirect tmpdir for modules being tested
36a84b0 
The test for some modules write into `os.tmpdir()` and do not always
clean up afterwards which can fill the temporary directory on our CI.
Redirect the temporary directory for modules being tested into CITGM's
`tmpDir` which gets removed at the end of the run.

Refs: nodejs/build#1757
@richardlau richardlau mentioned this issue Apr 12, 2019
Merged
3 tasks
richardlau added a commit to richardlau/citgm that referenced this issue Apr 13, 2019
@richardlau
lib: redirect tmpdir for modules being tested
1e22464 
The test for some modules write into `os.tmpdir()` and do not always
clean up afterwards which can fill the temporary directory on our CI.
Redirect the temporary directory for modules being tested into CITGM's
`tmpDir` which gets removed at the end of the run.

Refs: nodejs/build#1757
richardlau added a commit to nodejs/citgm that referenced this issue Apr 13, 2019
@richardlau
lib: redirect tmpdir for modules being tested
4974854 
The test for some modules write into `os.tmpdir()` and do not always
clean up afterwards which can fill the temporary directory on our CI.
Redirect the temporary directory for modules being tested into CITGM's
`tmpDir` which gets removed at the end of the run.

Refs: nodejs/build#1757
@sam-github
Copy link
Contributor

@mhdawson yes, I deleted the foo-* directories in /tmp. This should be closed, IMO (I don't have permissions to do it myself).

@sam-github sam-github mentioned this issue Apr 16, 2019
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@sam-github @richardlau @mhdawson