Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

You are running out of disk space. The runner will stop working when the machine runs out of disk space. Free space left: 72 MB. How to fix this ? #10386

Closed
1 of 13 tasks
sagarkalankar1 opened this issue Aug 4, 2024 · 20 comments

Comments

@sagarkalankar1
Copy link

Description

  • I am using 'ubuntu-latest' as github runner for my pipeline.

  • My GitHub Actions pipeline defines a workflow for building and publishing an Android project to Firebase App Distribution.

  • My pipeline sometimes given warning - "You are running out of disk space. The runner will stop working when the machine runs out of disk space. Free space left: 72 MB" and pipeline fails.

  • Seeing other github issues similar to this one, I found and tried a workaround as below:-

sudo rm -rf /usr/share/dotnet
sudo rm -rf /opt/ghc
sudo rm -rf "/usr/local/share/boost"
sudo rm -rf "$AGENT_TOOLSDIRECTORY"

But above folders does not exist when I tried viewing folders on my github runner 'ubuntu-latest'.

I can see below files and folders:-
36G /usr
23G /usr/local
13G /usr/local/lib
12G /usr/local/lib/android/sdk
12G /usr/local/lib/android
12G /opt
8.5G /opt/hostedtoolcache
7.8G /usr/local/lib/android/sdk/ndk
6.7G /usr/lib
5.6G /usr/share
5.5G /usr/local/.ghcup/ghc
5.5G /usr/local/.ghcup
5.0G /opt/hostedtoolcache/CodeQL/2.18.1/x64/codeql
5.0G /opt/hostedtoolcache/CodeQL/2.18.1/x64
5.0G /opt/hostedtoolcache/CodeQL/2.18.1
5.0G /opt/hostedtoolcache/CodeQL
3.1G /swapfile
3.0G /opt/hostedtoolcache/CodeQL/2.18.1/x64/codeql/swift
2.8G /usr/local/.ghcup/ghc/9.10.1

What can I do for this issue ?

Platforms affected

  • Azure DevOps
  • GitHub Actions - Standard Runners
  • GitHub Actions - Larger Runners

Runner images affected

  • Ubuntu 20.04
  • Ubuntu 22.04
  • Ubuntu 24.04
  • macOS 12
  • macOS 13
  • macOS 13 Arm64
  • macOS 14
  • macOS 14 Arm64
  • Windows Server 2019
  • Windows Server 2022

Image version and build link

Runner version: '2.317.0'

Is it regression?

It is occuring randomly.

Expected behavior

It should not give this warning and build should be completed successfully and build should not fail due to disk space issue.

Actual behavior

My pipeline sometimes gives warning - "You are running out of disk space. The runner will stop working when the machine runs out of disk space. Free space left: 72 MB" and pipeline fails.

Repro steps

This issue comes due to disk space issue of github runner.

@Prabhatkumar59
Copy link

Hi @sagarkalankar1 -Thank you for bringing this issue to our attention. We will update you on this issue after investigating.

@danibonilha
Copy link

I'm facing a similar issue with an android build that was working fine last week. Now it fails with java.io.IOException: No space left on device

@Prabhatkumar59
Copy link

Prabhatkumar59 commented Aug 5, 2024

Hi @sagarkalankar1 - You can try the following strategies to free up space and optimize your pipeline:

A. Clean Up Unnecessary Directories and Files
You can add steps in your workflow to remove unnecessary directories and files to free up space.

name: Build and Publish Android Project

on:
  push:
    branches:
      - main
  pull_request:
    branches:
      - main

jobs:
  build:
    runs-on: ubuntu-latest

    steps:
    - name: Checkout repository
      uses: actions/checkout@v2

    - name: Set up JDK
      uses: actions/setup-java@v2
      with:
        distribution: 'adopt'
        java-version: '11'

    - name: Remove unnecessary directories to free up space
      run: |
        sudo rm -rf /usr/local/.ghcup
        sudo rm -rf /opt/hostedtoolcache/CodeQL
        sudo rm -rf /usr/local/lib/android/sdk/ndk
        sudo rm -rf /usr/share/dotnet
        sudo rm -rf /opt/ghc
        sudo rm -rf /usr/local/share/boost

    - name: Install dependencies
      run: |
        sudo apt-get update
        sudo apt-get install -y your-required-packages

    - name: Build the Android project
      run: ./gradlew build

    - name: Publish to Firebase App Distribution
      run: |
        # Your commands to publish the build to Firebase


B. **Cache Dependencies**
Use caching to avoid re-downloading dependencies, which can save space and speed up builds.

- name: Cache Gradle files
      uses: actions/cache@v2
      with:
        path: ~/.gradle/caches
        key: ${{ runner.os }}-gradle-${{ hashFiles('**/*.gradle*', '**/gradle-wrapper.properties') }}
        restore-keys: |
          ${{ runner.os }}-gradle-

C. **Optimize Gradle Configuration**

Optimize your Gradle configuration to reduce the amount of data generated during builds.

Add the following to your **gradle.properties**:

org.gradle.daemon=false
org.gradle.parallel=false
org.gradle.configureondemand=false

D. If disk space continues to be an issue, consider using self-hosted runners with larger disk space.

E. **Monitor Disk Usage**
Add steps to monitor disk usage throughout the build process.

- name: Check disk usage
      run: df -h

    - name: Check directory sizes
      run: du -sh /usr/local/* /opt/* /usr/* | sort -hr


F. **For your understanding I am providing an example of the workflow with added cleanup steps**:-

name: Build and Publish Android Project

on:
  push:
    branches:
      - main
  pull_request:
    branches:
      - main

jobs:
  build:
    runs-on: ubuntu-latest

    steps:
    - name: Checkout repository
      uses: actions/checkout@v2

    - name: Set up JDK
      uses: actions/setup-java@v2
      with:
        distribution: 'adopt'
        java-version: '11'

    - name: Remove unnecessary directories to free up space
      run: |
        sudo rm -rf /usr/local/.ghcup
        sudo rm -rf /opt/hostedtoolcache/CodeQL
        sudo rm -rf /usr/local/lib/android/sdk/ndk
        sudo rm -rf /usr/share/dotnet
        sudo rm -rf /opt/ghc
        sudo rm -rf /usr/local/share/boost

    - name: Install dependencies
      run: |
        sudo apt-get update
        sudo apt-get install -y your-required-packages

    - name: Build the Android project
      run: ./gradlew build

    - name: Publish to Firebase App Distribution
      run: |
        # Your commands to publish the build to Firebase

    - name: Check disk usage
      run: df -h

    - name: Check directory sizes
      run: du -sh /usr/local/* /opt/* /usr/* | sort -hr

By using above steps, you can try to manage disk space and reduce the likelihood of your builds failing due to insufficient disk space. 







@danibonilha
Copy link

I'm facing a similar issue with an android build that was working fine last week. Now it fails with java.io.IOException: No space left on device

The issue seems to be with the ubuntu-latest image since the pipeline works if I run on an mac-os image and it was working before the latest update.

Was working:
Image: ubuntu-22.04
Version: 20240721.1.0

Is failing:
Image: ubuntu-22.04
Version: 20240730.2.0

@mantljosh
Copy link

We are running into a similar issue - we run elasticsearch in docker that we use in our test suite, and we get an error high disk watermark [90%] exceeded which prevents it from working. We started observing this when the image changed to 20240730.2.0 - last successful run was on 20240721.1.0

@alainib
Copy link

alainib commented Aug 6, 2024

same probleme for me

Run df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/root         73G   4G    8.7G  89%  /
tmpfs              3.9G  172K  3.9G   1% /dev/shm
tmpfs              1.6G  1.1M  1.6G   1% /run
tmpfs              5.0M     0  5.0M   0% /run/lock
/dev/sda15      105M  6.1M   99M   6% /boot/efi
tmpfs               93M   12K  793M   1% /run/user/1001

i just run clean action before with this script

name: Free Disk Space (Android)

on:
  workflow_dispatch: # To trigger manually

jobs:
  free-disk-space:
    runs-on: ubuntu-latest
    steps:
      - name: Free Disk Space (Ubuntu)
        uses: jlumbroso/free-disk-space@main
        with:
          tool-cache: false
 

any solution please ?

@Prabhatkumar59
Copy link

@alainib @danibonilha @mantljosh - By this week, new image version will be rolled out. So, kindly try once after new image version rolled out.
However, for you, I am providing an updated workflow that integrates more aggressive cleanup steps:-

name: Build and Publish Android Project

on:
  push:
    branches:
      - main
  pull_request:
    branches:
      - main

jobs:
  build:
    runs-on: ubuntu-22.04  # Specify a fixed version of Ubuntu

    steps:
    - name: Checkout repository
      uses: actions/checkout@v2

    - name: Free Disk Space Before Build
      run: |
        echo "Disk space before cleanup:"
        df -h
        sudo rm -rf /usr/local/.ghcup
        sudo rm -rf /opt/hostedtoolcache/CodeQL
        sudo rm -rf /usr/local/lib/android/sdk/ndk
        sudo rm -rf /usr/share/dotnet
        sudo rm -rf /opt/ghc
        sudo rm -rf /usr/local/share/boost
        sudo apt-get clean
        echo "Disk space after cleanup:"
        df -h

    - name: Set up JDK
      uses: actions/setup-java@v2
      with:
        distribution: 'adopt'
        java-version: '11'

    - name: Install dependencies
      run: |
        sudo apt-get update
        sudo apt-get install -y your-required-packages

    - name: Cache Gradle files
      uses: actions/cache@v2
      with:
        path: ~/.gradle/caches
        key: ${{ runner.os }}-gradle-${{ hashFiles('**/*.gradle*', '**/gradle-wrapper.properties') }}
        restore-keys: |
          ${{ runner.os }}-gradle-

    - name: Build the Android project
      run: ./gradlew build

    - name: Publish to Firebase App Distribution
      run: |
        # Your commands to publish the build to Firebase

    - name: Free Disk Space After Build
      run: |
        echo "Disk space before post-build cleanup:"
        df -h
        sudo rm -rf /usr/local/.ghcup
        sudo rm -rf /opt/hostedtoolcache/CodeQL
        sudo rm -rf /usr/local/lib/android/sdk/ndk
        sudo rm -rf /usr/share/dotnet
        sudo rm -rf /opt/ghc
        sudo rm -rf /usr/local/share/boost
        sudo apt-get clean
        echo "Disk space after post-build cleanup:"
        df -h

Explanation:-
Pre-Build Cleanup:

The script cleans up various large directories (/usr/local/.ghcup, /opt/hostedtoolcache/CodeQL, /usr/local/lib/android/sdk/ndk, /usr/share/dotnet, /opt/ghc, /usr/local/share/boost) to free up space.
df -h is used to show the disk space before and after cleanup for monitoring purposes.
Post-Build Cleanup:

Similar cleanup steps are performed after the build to ensure any temporary files created during the build process are removed.
df -h is again used to show the disk space before and after cleanup.

By this approach, you should be able to manage disk space more effectively and prevent build failures due to insufficient disk space.

@alainib
Copy link

alainib commented Aug 6, 2024

@Prabhatkumar59 wow thank you for fast answer and response :)

i will try it . what is the best approach please ?

  1. add this to the yml script who fail and run it each time
  2. create another yaml file and run it only when needed

i'm not sure if multiples yml file share folders like in macos/linux etc or each one have it's one space like in docker and running cleanup yml file whould not affect my reel yml file . thanks

@mantljosh
Copy link

FWIW we changed ubuntu-latest to ubuntu-24.04 and it seems to be working okay with the new version

@Prabhatkumar59
Copy link

Prabhatkumar59 commented Aug 6, 2024

Hi @alainib - Here are the best approaches which I am providing you to handle the disk space issue effectively:

Approach 1: Add Cleanup Steps to the Failing YAML File
Adding cleanup steps directly to the failing YAML file ensures that cleanup is performed every time the workflow runs. This approach guarantees that disk space issues are addressed before the build process starts.

Updated YAML File Example:

name: Build and Publish Android Project

on:
  push:
    branches:
      - main
  pull_request:
    branches:
      - main

jobs:
  build:
    runs-on: ubuntu-latest

    steps:
    - name: Checkout repository
      uses: actions/checkout@v2

    - name: Free Disk Space Before Build
      run: |
        echo "Disk space before cleanup:"
        df -h
        sudo rm -rf /usr/local/.ghcup
        sudo rm -rf /opt/hostedtoolcache/CodeQL
        sudo rm -rf /usr/local/lib/android/sdk/ndk
        sudo rm -rf /usr/share/dotnet
        sudo rm -rf /opt/ghc
        sudo rm -rf /usr/local/share/boost
        sudo apt-get clean
        echo "Disk space after cleanup:"
        df -h

    - name: Set up JDK
      uses: actions/setup-java@v2
      with:
        distribution: 'adopt'
        java-version: '11'

    - name: Install dependencies
      run: |
        sudo apt-get update
        sudo apt-get install -y your-required-packages

    - name: Cache Gradle files
      uses: actions/cache@v2
      with:
        path: ~/.gradle/caches
        key: ${{ runner.os }}-gradle-${{ hashFiles('**/*.gradle*', '**/gradle-wrapper.properties') }}
        restore-keys: |
          ${{ runner.os }}-gradle-

    - name: Build the Android project
      run: ./gradlew build

    - name: Publish to Firebase App Distribution
      run: |
        # Your commands to publish the build to Firebase

    - name: Free Disk Space After Build
      run: |
        echo "Disk space before post-build cleanup:"
        df -h
        sudo rm -rf /usr/local/.ghcup
        sudo rm -rf /opt/hostedtoolcache/CodeQL
        sudo rm -rf /usr/local/lib/android/sdk/ndk
        sudo rm -rf /usr/share/dotnet
        sudo rm -rf /opt/ghc
        sudo rm -rf /usr/local/share/boost
        sudo apt-get clean
        echo "Disk space after post-build cleanup:"
        df -h

This approach 1 ensures that cleanup is performed before every build, preventing disk space issues from affecting the build process.

Approach 2: Create a Separate Cleanup YAML File

You can try to create a separate YAML file dedicated to disk space cleanup. This approach allows you to trigger the cleanup workflow manually or on a schedule, without cluttering the main build YAML file.

Cleanup YAML File Example:

name: Free Disk Space

on:
  workflow_dispatch:  # Trigger manually
  schedule:
    - cron: '0 0 * * *'  # Runs daily at midnight

jobs:
  cleanup:
    runs-on: ubuntu-latest

    steps:
    - name: Free Disk Space
      run: |
        echo "Disk space before cleanup:"
        df -h
        sudo rm -rf /usr/local/.ghcup
        sudo rm -rf /opt/hostedtoolcache/CodeQL
        sudo rm -rf /usr/local/lib/android/sdk/ndk
        sudo rm -rf /usr/share/dotnet
        sudo rm -rf /opt/ghc
        sudo rm -rf /usr/local/share/boost
        sudo apt-get clean
        echo "Disk space after cleanup:"
        df -h

Main Build YAML File (Simplified) which is as follows:-

name: Build and Publish Android Project

on:
  push:
    branches: [ main ]
  pull_request:
    branches: [ main ]

jobs:
  build:
    runs-on: ubuntu-latest

    steps:
    - name: Checkout repository
      uses: actions/checkout@v2

    - name: Set up JDK
      uses: actions/setup-java@v2
      with:
        distribution: 'adopt'
        java-version: '11'

    - name: Install dependencies
      run: |
        sudo apt-get update
        sudo apt-get install -y your-required-packages

    - name: Cache Gradle files
      uses: actions/cache@v2
      with:
        path: ~/.gradle/caches
        key: ${{ runner.os }}-gradle-${{ hashFiles('**/*.gradle*', '**/gradle-wrapper.properties') }}
        restore-keys: |
          ${{ runner.os }}-gradle-

    - name: Build the Android project
      run: ./gradlew build

    - name: Publish to Firebase App Distribution
      run: |
       # Your commands to publish the build to Firebase

This approach 2 provides flexibility, allowing you to trigger cleanup manually or on a schedule. This keeps the main build YAML file clean and focused.

So, by using the the above approaches, you will able to fix those issues.

@Prabhatkumar59
Copy link

@sagarkalankar1 - I hope this issue is resolved and we are closing this issue. Feel free to reach out to us...thanks

@Cavaler
Copy link

Cavaler commented Aug 14, 2024

I'm still experiencing the same issue, even with the (seemingly) new image:

Current runner version: '2.319.0'
Operating System
  Ubuntu
  22.04.4
  LTS
Runner Image
  Image: ubuntu-22.04
  Version: 20240804.1.0
  Included Software: https://github.com/actions/runner-images/blob/ubuntu22/20240804.1/images/ubuntu/Ubuntu2204-Readme.md
  Image Release: https://github.com/actions/runner-images/releases/tag/ubuntu22%2F20240804.1
Runner Image Provisioner
  2.0.374.1

@Cavaler
Copy link

Cavaler commented Aug 14, 2024

The cleanup procedure from above raises free space from 16 to 36Gb.

@HinTak
Copy link

HinTak commented Aug 15, 2024

I cannot do any clean-up steps since the role of the runner image is simply to load another linux's docker container. The free space seems to have gotten lower in the last 6 to 12 months from about 26GB down to 19GB at the moment. It was 21GB 3 weeks ago. I have a df at the beginning of the start of the container.

Switching from ubuntu-latest (22.04) to ubuntu-24.04 explicitly, gives me an extra 10GB from 19GB back to 30GB.

I am wondering if the apt-get clean should get done at the end of the preparation of the runner image itself, since the runner image seems to have gotten gradually worse over time.
sudo apt-get clean

@HinTak
Copy link

HinTak commented Aug 15, 2024

I am doing

    runs-on: ubuntu-24.04 # unbuntu-latest is 22.04
    container: fedora:40

so any clean-up of ubuntu can't be done.

@BrentMifsud
Copy link

BrentMifsud commented Aug 31, 2024

getting this issue on the m1 Mac runners as well. Only started a few hours ago.

There have been no changes on our end. Our jobs just randomly started failing because of this issue.

@paulb777
Copy link

Also seeing this in Firebase builds on the macos-14 runner starting Friday, August 30, on otherwise unchanged configurations - https://github.com/firebase/firebase-ios-sdk/actions/runs/10626580586

@exceptionalvic
Copy link

@alainib @danibonilha @mantljosh - By this week, new image version will be rolled out. So, kindly try once after new image version rolled out. However, for you, I am providing an updated workflow that integrates more aggressive cleanup steps:-

name: Build and Publish Android Project

on:
  push:
    branches:
      - main
  pull_request:
    branches:
      - main

jobs:
  build:
    runs-on: ubuntu-22.04  # Specify a fixed version of Ubuntu

    steps:
    - name: Checkout repository
      uses: actions/checkout@v2

    - name: Free Disk Space Before Build
      run: |
        echo "Disk space before cleanup:"
        df -h
        sudo rm -rf /usr/local/.ghcup
        sudo rm -rf /opt/hostedtoolcache/CodeQL
        sudo rm -rf /usr/local/lib/android/sdk/ndk
        sudo rm -rf /usr/share/dotnet
        sudo rm -rf /opt/ghc
        sudo rm -rf /usr/local/share/boost
        sudo apt-get clean
        echo "Disk space after cleanup:"
        df -h

    - name: Set up JDK
      uses: actions/setup-java@v2
      with:
        distribution: 'adopt'
        java-version: '11'

    - name: Install dependencies
      run: |
        sudo apt-get update
        sudo apt-get install -y your-required-packages

    - name: Cache Gradle files
      uses: actions/cache@v2
      with:
        path: ~/.gradle/caches
        key: ${{ runner.os }}-gradle-${{ hashFiles('**/*.gradle*', '**/gradle-wrapper.properties') }}
        restore-keys: |
          ${{ runner.os }}-gradle-

    - name: Build the Android project
      run: ./gradlew build

    - name: Publish to Firebase App Distribution
      run: |
        # Your commands to publish the build to Firebase

    - name: Free Disk Space After Build
      run: |
        echo "Disk space before post-build cleanup:"
        df -h
        sudo rm -rf /usr/local/.ghcup
        sudo rm -rf /opt/hostedtoolcache/CodeQL
        sudo rm -rf /usr/local/lib/android/sdk/ndk
        sudo rm -rf /usr/share/dotnet
        sudo rm -rf /opt/ghc
        sudo rm -rf /usr/local/share/boost
        sudo apt-get clean
        echo "Disk space after post-build cleanup:"
        df -h

Explanation:- Pre-Build Cleanup:

The script cleans up various large directories (/usr/local/.ghcup, /opt/hostedtoolcache/CodeQL, /usr/local/lib/android/sdk/ndk, /usr/share/dotnet, /opt/ghc, /usr/local/share/boost) to free up space. df -h is used to show the disk space before and after cleanup for monitoring purposes. Post-Build Cleanup:

Similar cleanup steps are performed after the build to ensure any temporary files created during the build process are removed. df -h is again used to show the disk space before and after cleanup.

By this approach, you should be able to manage disk space more effectively and prevent build failures due to insufficient disk space.

works!

@liudonghua123
Copy link

I have the similar problems. Why not increase the disk size of action runner to about 50G free.

@liudonghua123
Copy link

Prabhatkumar59

@Prabhatkumar59 Hi, I didn't understand your approach 2, when the main build job trigger cleanup job? I didn't find the connection between them. Could you explain more in details.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests