Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

GitHub Actions failes with: "no space left on device" #2875

Closed
1 of 7 tasks
Imipenem opened this issue Mar 8, 2021 · 6 comments
Closed
1 of 7 tasks

GitHub Actions failes with: "no space left on device" #2875

Imipenem opened this issue Mar 8, 2021 · 6 comments
Assignees
@al-cheb
Labels
Area: Scripting and command line OS: Ubuntu

Comments

@Imipenem
Copy link

Imipenem commented Mar 8, 2021 •
edited
Loading

Description
A clear and concise description of what the bug is, and why you consider it to be a bug.
I recently noticed that one of my GitHub workflows failed with the above mentioned error.
Link to one of those runs: https://github.com/Imipenem/mypytest/runs/2060441074.

i followed #709 and added:
sudo rm -rf "/usr/local/share/boost"
sudo rm -rf "$AGENT_TOOLSDIRECTORY"

But this lead to the error that I could not execute some python packages etc which is crucial for my workflow.

So I decided to remove some space consuming packages, but the error still persists.

Area for Triage:

Question, Bug, or Feature?:

Bug

Virtual environments affected

  • Ubuntu 16.04
  • Ubuntu 18.04
  • Ubuntu 20.04
  • macOS 10.15
  • macOS 11.0
  • Windows Server 2016 R2
  • Windows Server 2019

Image version

Image version where you are experiencing the issue.
Version: 20210302.0

Expected behavior
A description of what you expected to happen.
The workflow should pass.

Actual behavior
A description of what is actually happening.
Workflow keeps failing with running out of space

Repro steps
A description with steps to reproduce the issue. If your have a public example or repo to share,
please provide the link.

  1. https://github.com/Imipenem/mypytest/runs/2060441074 link for example WF run (accidently ran WF another time, so link is not the initial run)

Exact error:
Error processing tar file(exit status 1): write /home/user/miniconda/envs/mypytest/lib/python3.9/site-packages/torch/lib/libtorch_cuda_cu.so: no space left on device Error: exit status 1

Any ideas @miketimofeev or others?

Thanks ;)
@Imipenem Imipenem mentioned this issue Mar 8, 2021
Merged
@al-cheb
Copy link
Contributor

al-cheb commented Mar 9, 2021

Hello, @Imipenem
I think you can delete donet&ghc locations - #2840 (comment)

@Imipenem
Copy link
Author

Imipenem commented Mar 9, 2021

Thanks @al-cheb will try this

@Imipenem
Copy link
Author

Imipenem commented Mar 9, 2021

This worked, thanks @al-cheb

@Imipenem Imipenem closed this as completed Mar 9, 2021
@jlumbroso
Copy link

@Imipenem Having faced this siutation so many times, I decided to stop rewriting the wheel 😁 and wrote a GitHub Actions to do this properly:

https://github.com/marketplace/actions/free-disk-space-ubuntu

Open to suggestions. I noticed these lines:

sudo rm -rf "/usr/local/share/boost"
sudo rm -rf "$AGENT_TOOLSDIRECTORY"

but I am not sure what they remove?

@miketimofeev
Copy link
Contributor

@jlumbroso the first one doesn't exist anymore on the image. The second one removes all the pre-cached tools (Node, Go, Python, Ruby) that are used by the correspondent actions like https://github.com/actions/setup-python

@jlumbroso
Copy link

Thanks so much @miketimofeev! Thanks to you, I have added an option tool-cache that removes the pre-cached tools, it saves an extra 6GB. You also taught me how to look at the internals of GitHub Actions runners 😁 🙏🏻

https://github.com/jlumbroso/free-disk-space/releases/tag/v1.1.0

lorisleiva added a commit to metaplex-foundation/mpl-token-auth-rules that referenced this issue May 18, 2023
@lorisleiva
Use buildjet runner when testing programs
b1aebe2 
Without it we're getting a "no space left on device" error.

See actions/runner-images#2875
lorisleiva added a commit to metaplex-foundation/mpl-token-auth-rules that referenced this issue May 18, 2023
@lorisleiva
Add Umi library and reshape the repository (#128)
d013486 
* Move program folder to programs/token-auth-rules

* Rename packages/sdk to clients/js-solita

* Move cli into clients

* Add configs folder

* Remove old JS configs

* Remove old scripts

* Update package.json and replace yarn with pnpm

* Update GitHub workflows

* Format config folder

* Add Amman client back to JS Solita client

* Add Umi JS client

* Regenerate client with correct program ID

* Add .env file from GitHub actions

* Fix CLI dependencies

* Enable Umi library publishing on NPM

* Use buildjet runner when testing programs

Without it we're getting a "no space left on device" error.

See actions/runner-images#2875
@ManavalanG ManavalanG mentioned this issue Jun 20, 2023
Closed
@alexandrecuer alexandrecuer mentioned this issue Jul 27, 2023
Open
@v1v v1v mentioned this issue Aug 7, 2023
Merged
19 tasks
@realshuting realshuting mentioned this issue Sep 12, 2023
Merged
@hardfist hardfist mentioned this issue Oct 8, 2023
Merged
2 tasks
@Bchass Bchass mentioned this issue Oct 13, 2023
Merged
akoshelev added a commit to akoshelev/raw-ipa that referenced this issue Nov 8, 2023
@akoshelev
Clean up runner space before running actions
8192422 
14Gb default space is probably not enough for us anymore

```
[ec2-user@ip-172-31-49-57 ipa]$ du -sh target/
36G     target/
```

see the discussion here
actions/runner-images#2875
akoshelev added a commit to akoshelev/raw-ipa that referenced this issue Nov 8, 2023
@akoshelev
Clean up runner space before running actions
139ebc5 
14Gb default space is probably not enough for us anymore

```
[ec2-user@ip-172-31-49-57 ipa]$ du -sh target/
36G     target/
```

see the discussion here
actions/runner-images#2875
akoshelev added a commit to akoshelev/raw-ipa that referenced this issue Nov 8, 2023
@akoshelev
Clean up runner space before running actions
f78c6cb 
14Gb default space is probably not enough for us anymore

```
[ec2-user@ip-172-31-49-57 ipa]$ du -sh target/
36G     target/
```

see the discussion here
actions/runner-images#2875
@akoshelev akoshelev mentioned this issue Nov 8, 2023
Merged
akoshelev added a commit to private-attribution/ipa that referenced this issue Nov 9, 2023
@akoshelev
Clean up runner space before running actions
2fd0567 
14Gb default space is probably not enough for us anymore

```
[ec2-user@ip-172-31-49-57 ipa]$ du -sh target/
36G     target/
```

see the discussion here
actions/runner-images#2875
@alkidbaci alkidbaci mentioned this issue Dec 7, 2023
Closed
@gitlawr gitlawr mentioned this issue Feb 4, 2024
Merged
@sjain-stanford sjain-stanford mentioned this issue Feb 12, 2024
Merged
sjain-stanford added a commit to cruise-automation/mlir-tcp that referenced this issue Feb 13, 2024
@sjain-stanford
Disable third party builds (on push to main) until we migrate to self…
1bec376 
…-hosted runners (#38)

We're hitting `no space left on device` errors during llvm and stablehlo
builds with the free GitHub hosted runners, which come with limited disk
space allocations. Cleaning up the runners as recommended
[here](https://github.com/orgs/community/discussions/25678),
[here](https://github.com/orgs/community/discussions/26351) and
actions/runner-images#2875 have helped
partially (increase root mount from 20G to 47G) but it still isn't
enough for some builds. Temporarily disable the 3p builds on push to
main (they will still run as non-merge gating on PRs), and re-enable
once self-hosted runners are setup.
@jorendumoulin jorendumoulin mentioned this issue Feb 13, 2024
Closed
@Br1ght0ne Br1ght0ne mentioned this issue Feb 13, 2024
Closed
@shivaraj-bh shivaraj-bh mentioned this issue Mar 6, 2024
Merged
@DhanshreeA DhanshreeA mentioned this issue Mar 13, 2024
Closed
@garysassano garysassano mentioned this issue Apr 28, 2024
Closed
@oscar-izval oscar-izval mentioned this issue Jul 17, 2024
Closed
@clemlesne clemlesne mentioned this issue Sep 1, 2024
Closed
@Chris-Petty Chris-Petty mentioned this issue Sep 2, 2024
Merged
@emiliom emiliom mentioned this issue Nov 19, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@al-cheb al-cheb

Labels
Area: Scripting and command line OS: Ubuntu
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@jlumbroso @Imipenem @al-cheb @miketimofeev @Darleev