Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

cross-compiler-rhel8-armv7-gcc-8-glibc-2.28: No space left on device #2977

Closed
F3n67u opened this issue Jun 27, 2022 · 5 comments · Fixed by #2978
Closed

cross-compiler-rhel8-armv7-gcc-8-glibc-2.28: No space left on device #2977

F3n67u opened this issue Jun 27, 2022 · 5 comments · Fixed by #2978
Labels
ci-public incident platform:arm platform:windows

Comments

@F3n67u
Copy link
Member

F3n67u commented Jun 27, 2022

Build link

https://ci.nodejs.org/job/node-cross-compile/38558/nodes=cross-compiler-rhel8-armv7-gcc-8-glibc-2.28/console

Console Output

15:27:11 + git checkout -B jenkins-node-test-commit-arm-fanned-ce986803d205c0ce68a618e54f3dfadf570b5477-binary-pi1p/cc-armv7
15:27:12 Switched to a new branch 'jenkins-node-test-commit-arm-fanned-ce986803d205c0ce68a618e54f3dfadf570b5477-binary-pi1p/cc-armv7'
15:27:12 A	binary/binary.tar.xz
15:27:12 + git config --replace-all user.name 'Node.js Jenkins CI'
15:27:12 + git config --replace-all user.email ci@iojs.org
15:27:12 + git commit -m 'added binaries'
15:27:12 [jenkins-node-test-commit-arm-fanned-ce986803d205c0ce68a618e54f3dfadf570b5477-binary-pi1p/cc-armv7 cc6333ff29] added binaries
15:27:12  1 file changed, 0 insertions(+), 0 deletions(-)
15:27:12  create mode 100644 binary/binary.tar.xz
15:27:12 + ssh-agent sh -c 'ssh-add **** && git push binary_tmp@147.75.66.203:binary_tmp.git +jenkins-node-test-commit-arm-fanned-ce986803d205c0ce68a618e54f3dfadf570b5477-binary-pi1p/cc-armv7'
15:27:12 Identity added: **** (****)
15:27:13 remote: error: file write error: No space left on device        
15:27:14 remote: fatal: unable to write sha1 file        
15:27:14 error: remote unpack failed: unpack-objects abnormal exit
15:27:14 To 147.75.66.203:binary_tmp.git
15:27:14  ! [remote rejected]       jenkins-node-test-commit-arm-fanned-ce986803d205c0ce68a618e54f3dfadf570b5477-binary-pi1p/cc-armv7 -> jenkins-node-test-commit-arm-fanned-ce986803d205c0ce68a618e54f3dfadf570b5477-binary-pi1p/cc-armv7 (unpacker error)
15:27:14 error: failed to push some refs to '147.75.66.203:binary_tmp.git'
15:27:14 Build step 'Execute shell' marked build as failure
15:27:15 Skipped archiving because build is not successful
15:27:15 Collecting metadata...
15:27:15 Metadata collection done.
15:27:15 Notifying upstream projects of job completion
15:27:15 Finished: FAILURE
@richardlau
Copy link
Member

The Windows builds are failing with similar errors (nodejs/node#43420 (comment)) -- it looks like the server hosting the temporary binary_tmp git repository used by the Windows and arm-fanned jobs has run out of space.

@F3n67u
Copy link
Member Author

F3n67u commented Jun 27, 2022

The Windows builds are failing with similar errors (nodejs/node#43420 (comment)) -- it looks like the server hosting the temporary binary_tmp git repository used by the Windows and arm-fanned jobs has run out of space.

Could you please take a look at? At least 3 pr I have seen failed with this problem.

@F3n67u F3n67u mentioned this issue Jun 27, 2022
Merged
@richardlau
Copy link
Member

The Windows builds are failing with similar errors (nodejs/node#43420 (comment)) -- it looks like the server hosting the temporary binary_tmp git repository used by the Windows and arm-fanned jobs has run out of space.

Could you please take a look at? At least 3 pr I have seen failed with this problem.

I'm trying to figure this out now.

@richardlau
Copy link
Member 

root@test-packetnet-ubuntu1804-x64-1:~# df -h
Filesystem      Size  Used Avail Use% Mounted on
udev             16G     0   16G   0% /dev
tmpfs           3.2G  342M  2.8G  11% /run
/dev/md127      218G  207G  4.5M 100% /
tmpfs            16G     0   16G   0% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs            16G     0   16G   0% /sys/fs/cgroup
tmpfs           3.2G     0  3.2G   0% /run/user/0
root@test-packetnet-ubuntu1804-x64-1:~# du -hs /home/iojs/*
199G    /home/iojs/build
628M    /home/iojs/git
20M     /home/iojs/jenkins_diagnostics.txt
1.5M    /home/iojs/slave.jar
4.0K    /home/iojs/tmp
root@test-packetnet-ubuntu1804-x64-1:~# du -hs /home/iojs/build/*
79G     /home/iojs/build/binary_tmp.git
8.0K    /home/iojs/build/caches
11M     /home/iojs/build/tools
120G    /home/iojs/build/workspace
root@test-packetnet-ubuntu1804-x64-1:~#

From #2226 (comment) I think it should be safe to clear out /home/iojs/build/workspace (Jenkins should repopulate).

Also I wasn't sure if 79G was large for /home/iojs/build/binary_tmp.git -- looking at ~binary_tmp/clean_binary_tmp.log it looks like:

  • Usually the disk usage is between 30-38G after cleaning the binary tmp repository
  • More seriously it looks like the cleanup stopped cleaning after 15 June:
Wed Jun 15 05:00:01 UTC 2022
38G     /home/binary_tmp/binary_tmp.git/
Deleted branch jenkins-node-test-commit-windows-coverage-daily-aba2cd74dce406da2f3dcf6a2bd08d9552fa3671 (was aba2cd74dce4).
Deleted branch jenkins-node-test-commit-windows-fanned-2d287715a96448f4debc751e5606de9b00c7f6e0 (was 5c9054e38a4b).
Deleted branch jenkins-node-test-commit-windows-fanned-2d287715a96448f4debc751e5606de9b00c7f6e0-bin-win-vs2019 (was 3c6a6c222fcd).
Deleted branch jenkins-node-test-commit-windows-fanned-2d287715a96448f4debc751e5606de9b00c7f6e0-bin-win-vs2019-arm64 (was 86ab2370f762).
Deleted branch jenkins-node-test-commit-windows-fanned-2d287715a96448f4debc751e5606de9b00c7f6e0-bin-win-vs2019-x86 (was 906b51562dd0).
Deleted branch jenkins-node-test-commit-windows-fanned-2e521c0140a97db1a49ba237862dd5f3f17169cd (was e4e5ef34e97c).
Deleted branch jenkins-node-test-commit-windows-fanned-2e521c0140a97db1a49ba237862dd5f3f17169cd-bin-win-vs2019 (was 1eaa288414ad).
Deleted branch jenkins-node-test-commit-windows-fanned-2e521c0140a97db1a49ba237862dd5f3f17169cd-bin-win-vs2019-arm64 (was e96c5ac239e3).
Deleted branch jenkins-node-test-commit-windows-fanned-2e521c0140a97db1a49ba237862dd5f3f17169cd-bin-win-vs2019-x86 (was 7ea4dc084349).
Deleted branch jenkins-node-test-commit-windows-fanned-461e0e4e15fd49c695aeb6e7f8b91c949ad085e9 (was 461e0e4e15fd).
Deleted branch jenkins-node-test-commit-windows-fanned-aba2cd74dce406da2f3dcf6a2bd08d9552fa3671 (was aba2cd74dce4).
Deleted branch jenkins-node-test-commit-windows-fanned-b631086208ef5654a8f12fc8241db7dd46201eef-bin-win-vs2019 (was 8b4404694791).
Deleted branch jenkins-node-test-commit-windows-fanned-b631086208ef5654a8f12fc8241db7dd46201eef-bin-win-vs2019-arm64 (was 320a9f158a34).
Deleted branch jenkins-node-test-commit-windows-fanned-b631086208ef5654a8f12fc8241db7dd46201eef-bin-win-vs2019-x86 (was 980926421b94).
37G     /home/binary_tmp/binary_tmp.git/

Thu Jun 16 05:00:01 UTC 2022
41G     /home/binary_tmp/binary_tmp.git/

Fri Jun 17 05:00:01 UTC 2022
43G     /home/binary_tmp/binary_tmp.git/

Manually running the clean up script throws this error:

root@test-packetnet-ubuntu1804-x64-1:~# sudo -H -u binary_tmp ~binary_tmp/clean_binary_tmp.sh
+ cd /home/binary_tmp/binary_tmp.git
+ echo
+ date
+ du -sh /home/binary_tmp/binary_tmp.git/
+ git fetch origin +master:master
fatal: Couldn't find remote ref master
root@test-packetnet-ubuntu1804-x64-1:~#

which is related to the branch rename (nodejs/node#33864) -- looks like we missed another reference.

I'll recover some space -- give me a couple of hours (lunch and then work meeting).

@F3n67u F3n67u mentioned this issue Jun 27, 2022
Closed
@richardlau richardlau mentioned this issue Jun 27, 2022
Merged
@richardlau
Copy link
Member

I've updated the cleanup script (#2978), deployed via Ansible (had to fix one of the tasks to run as the owner of the git repository on the machine), and run the updated clean up script which has pruned the repository back to 30G.

I've also cleaned out /home/iojs/build/workspace and updated the bare reference repository (via git fetch origin) to speed the creation of the fresh workspaces.

root@test-packetnet-ubuntu1804-x64-1:~# df -h
Filesystem      Size  Used Avail Use% Mounted on
udev             16G     0   16G   0% /dev
tmpfs           3.2G  350M  2.8G  11% /run
/dev/md127      218G   39G  169G  19% /
tmpfs            16G     0   16G   0% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs            16G     0   16G   0% /sys/fs/cgroup
tmpfs           3.2G     0  3.2G   0% /run/user/0
root@test-packetnet-ubuntu1804-x64-1:~# du -hs /home/iojs/build/*
31G     /home/iojs/build/binary_tmp.git
8.0K    /home/iojs/build/caches
11M     /home/iojs/build/tools
24K     /home/iojs/build/workspace
root@test-packetnet-ubuntu1804-x64-1:~#

richardlau added a commit that referenced this issue Jun 28, 2022
@richardlau
ansible: update branch used in clean_binary_tmp.sh (#2978)
65b68ec 
Update `clean_binary_tmp.sh` script to reference `main` instead of
`master`.

Fix up the `jenkins-workspace` role to run as the owner of the git
repository when disabling automatic gc.

Fixes: #2977
Refs: nodejs/node#33864
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
ci-public incident platform:arm platform:windows
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

ansible: update branch used in clean_binary_tmp.sh richardlau/build
2 participants
@richardlau @F3n67u