Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug]: Used too much storage to store build intermediates #1161

Closed
3 tasks done
lihenggui opened this issue Jan 20, 2024 · 9 comments
Closed
3 tasks done

[Bug]: Used too much storage to store build intermediates #1161

lihenggui opened this issue Jan 20, 2024 · 9 comments
Labels
bug Something isn't working

Comments

@lihenggui
Copy link
Contributor

lihenggui commented Jan 20, 2024 •
edited
Loading

Is there an existing issue for this?

  • I have searched the existing issues

Is there a StackOverflow question about this issue?

  • I have searched StackOverflow

What happened?

Usually, I will run this command to check there is no compile error in the project.
./gradlew --init-script gradle/init.gradle.kts --no-configuration-cache --no-daemon clean spotlessApply assembleRelease testClasses assembleAndroidTest --warning-mode all

I noticed that after finishing compiling, the size of the project folder grows rapidly, it will use about 15GB of disk space to store build intermediates.
image
image

Each feature module took around 1.3G of disk space:
image
image
image

In the core folder, the size of designsystem, ui, and database modules are abnormal, for other modules, the size is ok.

I am curious if it is expected to use so much storage to save build intermediates for a single module. The size can be easily 'exploded' by adding more modules to the project.

GitHub-hosted free tier runners provide 14G of storage to run actions, the storage limit of the runners is almost the same as the size of the project. If we add one more module, the action will fail and throw a No space left error, causing the CI to fail all the time.

Relevant logcat output

No response

Code of Conduct

  • I agree to follow this project's Code of Conduct
@lihenggui lihenggui added the bug Something isn't working label Jan 20, 2024
@SimonMarquis
Copy link
Contributor

  1. When executing assembleRelease testClasses assembleAndroidTest you are effectively asking Gradle to build almost everything in all variants:
  • assembleRelease: assembleProdRelease/assembleDemoRelease for all Android modules
  • assembleAndroidTest: assembleDemoDebugAndroidTest/assembleProdDebugAndroidTest for all Android modules
  • testClasses: build JVM modules only
    You are therefore "wasting" a lot of compute time to build artifacts you probably don't need.
  1. Test APKs are bundled with a lot of junk (not sure what is including them at the moment, agp, build-tools, etc.?) and can be found in intermediates artifacts as well.
  • Fonts (~150M) (especially NotoSansCJK)
  • IBM's ICU (~50M)
  • librobolectric (~32M) (for macos while using windows...)
  • sqlite4java (~2.7M) (all arch)

If feel like this is not normal 🤯

Details of a test apk... 

.
├── [ 26M]  com
│   └── [ 26M]  ibm
│       └── [ 26M]  icu
│           ├── [2.9K]  ICUConfig.properties
│           └── [ 26M]  impl
├── [159M]  fonts
│   ├── [3.8M]  DroidSansFallback.ttf
│   ├── [1.2M]  NanumGothic.ttf
│   ├── [9.0M]  NotoColorEmoji.ttf
│   ├── [ 18M]  NotoSansCJK-Black.ttc
│   ├── [ 18M]  NotoSansCJK-Bold.ttc
│   ├── [ 17M]  NotoSansCJK-DemiLight.ttc
│   ├── [ 17M]  NotoSansCJK-Light.ttc
│   ├── [ 17M]  NotoSansCJK-Medium.ttc
│   ├── [ 18M]  NotoSansCJK-Regular.ttc
│   ├── [ 16M]  NotoSansCJK-Thin.ttc
│   └── ...
├── [ 25M]  icu
│   └── [ 25M]  icudt68l.dat
├── [ 32M]  native
│   └── [ 32M]  mac
│       ├── [ 15M]  aarch64
│       │   └── [ 15M]  librobolectric-nativeruntime.dylib
│       └── [ 16M]  x86_64
│           └── [ 16M]  librobolectric-nativeruntime.dylib
└── [2.7M]  sqlite4java
    ├── [1.5M]  osx
    │   └── [1.5M]  libsqlite4java.dylib
    ├── [668K]  win32-x64
    │   └── [668K]  sqlite4java.dll
    └── [518K]  win32-x86
        └── [517K]  sqlite4java.dll

@SimonMarquis
Copy link
Contributor

Found this issue 🕵️ preparing the fix with a report…

@lihenggui
Copy link
Contributor Author

  1. When executing assembleRelease testClasses assembleAndroidTest you are effectively asking Gradle to build almost everything in all variants:
    You are therefore "wasting" a lot of compute time to build artifacts you probably don't need.

Yes, I did it intentionally. I need to check if there are any compilation issues in the code. Sometimes I modify the ViewModel but Android Studio won't notify me if there are errors that exist in the ViewModelTests. So I have to compile everything to avoid failures in CI.

SimonMarquis added a commit to SimonMarquis/nowinandroid that referenced this issue Jan 20, 2024
@SimonMarquis
Extract Roborazzi utils into :roborazzi module
c091341 
And cleanup unnecessary transitive dependencies.

Fixes android#1161
@SimonMarquis SimonMarquis mentioned this issue Jan 20, 2024
Closed
@SimonMarquis
Copy link
Contributor

SimonMarquis commented Jan 20, 2024 •
edited
Loading

The main issue comes from robolectric being added to the runtime classpath of instrumented tests because it is transitively exported by :core:testing.

nowinandroid/core/testing/build.gradle.kts

Line 43 in aea26aa

implementation(libs.robolectric.shadows)

Here are some stats/results with different solutions:

  • Default (main branch)

    BUILD SUCCESSFUL in 17m 10s
2259 actionable tasks: 2256 executed, 3 up-to-date

du --human-readable --summarize .
7.6G    .

find . -name "*.apk" -exec du --human-readable --summarize '{}' \;
159M    ./app/build/outputs/apk/androidTest/demo/debug/app-demo-debug-androidTest.apk
23M     ./app/build/outputs/apk/demo/debug/app-demo-debug.apk
177M    ./core/database/build/outputs/apk/androidTest/demo/debug/database-demo-debug-androidTest.apk
177M    ./core/designsystem/build/outputs/apk/androidTest/demo/debug/designsystem-demo-debug-androidTest.apk
177M    ./core/ui/build/outputs/apk/androidTest/demo/debug/ui-demo-debug-androidTest.apk
177M    ./feature/bookmarks/build/outputs/apk/androidTest/demo/debug/bookmarks-demo-debug-androidTest.apk
177M    ./feature/foryou/build/outputs/apk/androidTest/demo/debug/foryou-demo-debug-androidTest.apk
177M    ./feature/interests/build/outputs/apk/androidTest/demo/debug/interests-demo-debug-androidTest.apk
177M    ./feature/search/build/outputs/apk/androidTest/demo/debug/search-demo-debug-androidTest.apk
178M    ./feature/settings/build/outputs/apk/androidTest/demo/debug/settings-demo-debug-androidTest.apk
177M    ./feature/topic/build/outputs/apk/androidTest/demo/debug/topic-demo-debug-androidTest.apk
177M    ./sync/work/build/outputs/apk/androidTest/demo/debug/work-demo-debug-androidTest.apk

  • 1st solution (changing the configuration) Remove unexpected libs.robolectric.shadows dependency SimonMarquis/nowinandroid#20

    -implementation(libs.robolectric.shadows)
+compileOnly(libs.robolectric.shadows)
    BUILD SUCCESSFUL in 13m 33s
2259 actionable tasks: 2254 executed, 5 up-to-date

du --human-readable --summarize .
2.3G    .

find . -name "*.apk" -exec du --human-readable --summarize '{}' \;
1.7M    ./app/build/outputs/apk/androidTest/demo/debug/app-demo-debug-androidTest.apk
23M     ./app/build/outputs/apk/demo/debug/app-demo-debug.apk
18M     ./core/database/build/outputs/apk/androidTest/demo/debug/database-demo-debug-androidTest.apk
19M     ./core/designsystem/build/outputs/apk/androidTest/demo/debug/designsystem-demo-debug-androidTest.apk
19M     ./core/ui/build/outputs/apk/androidTest/demo/debug/ui-demo-debug-androidTest.apk
19M     ./feature/bookmarks/build/outputs/apk/androidTest/demo/debug/bookmarks-demo-debug-androidTest.apk
19M     ./feature/foryou/build/outputs/apk/androidTest/demo/debug/foryou-demo-debug-androidTest.apk
19M     ./feature/interests/build/outputs/apk/androidTest/demo/debug/interests-demo-debug-androidTest.apk
19M     ./feature/search/build/outputs/apk/androidTest/demo/debug/search-demo-debug-androidTest.apk
20M     ./feature/settings/build/outputs/apk/androidTest/demo/debug/settings-demo-debug-androidTest.apk
19M     ./feature/topic/build/outputs/apk/androidTest/demo/debug/topic-demo-debug-androidTest.apk
20M     ./sync/work/build/outputs/apk/androidTest/demo/debug/work-demo-debug-androidTest.apk

  • 2nd solution 2 (:roborazzi) Extract Roborazzi utils into :roborazzi module #1163

    BUILD SUCCESSFUL in 11m 21s
2122 actionable tasks: 2118 executed, 4 up-to-date

du --human-readable --summarize .
2.0G    .

find . -name "*.apk" -exec du --human-readable --summarize '{}' \;
1.2M    ./app/build/outputs/apk/androidTest/demo/debug/app-demo-debug-androidTest.apk
23M     ./app/build/outputs/apk/demo/debug/app-demo-debug.apk
3.8M    ./core/database/build/outputs/apk/androidTest/demo/debug/database-demo-debug-androidTest.apk
17M     ./core/designsystem/build/outputs/apk/androidTest/demo/debug/designsystem-demo-debug-androidTest.apk
19M     ./core/ui/build/outputs/apk/androidTest/demo/debug/ui-demo-debug-androidTest.apk
19M     ./feature/bookmarks/build/outputs/apk/androidTest/demo/debug/bookmarks-demo-debug-androidTest.apk
19M     ./feature/foryou/build/outputs/apk/androidTest/demo/debug/foryou-demo-debug-androidTest.apk
19M     ./feature/interests/build/outputs/apk/androidTest/demo/debug/interests-demo-debug-androidTest.apk
19M     ./feature/search/build/outputs/apk/androidTest/demo/debug/search-demo-debug-androidTest.apk
20M     ./feature/settings/build/outputs/apk/androidTest/demo/debug/settings-demo-debug-androidTest.apk
19M     ./feature/topic/build/outputs/apk/androidTest/demo/debug/topic-demo-debug-androidTest.apk
7.4M    ./sync/work/build/outputs/apk/androidTest/demo/debug/work-demo-debug-androidTest.apk

@SimonMarquis SimonMarquis mentioned this issue Feb 19, 2024
Merged
@SimonMarquis
Copy link
Contributor

@lihenggui the main issue has been fixed during this PR:

I prepared a followup PR to clean things up a bit more in:

You can close the issue 👍

@lihenggui
Copy link
Contributor Author

@lihenggui the main issue has been fixed during this PR:

I prepared a followup PR to clean things up a bit more in:

You can close the issue 👍

Great, thank you!

@lihenggui
Copy link
Contributor Author

A question here, the dependency resolution behavior is reverted in the AGP 8.3.1 version.

The behavior change for library modules is reverting to 8.2 behavior in 8.4.0-alpha12, and this revert will also be cherry-picked to 8.3.x.

Does that mean this issue will appear in the latest version again?

@SimonMarquis
Copy link
Contributor

This was not the fix. The fix was mostly removing robolectric-shadow from the :core:testing module (which was included almost everywhere in the project)

https://github.com/android/nowinandroid/pull/1140/files#diff-6b9a7ef5da2d63d65731cee8c37d0228c481830ce6a813cdf9e9069c3cfd6288L43

@SimonMarquis
Copy link
Contributor

My PR #1163 was also fixing the issue, before the AGP update.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Extract Roborazzi utils into :roborazzi module SimonMarquis/nowinandroid
2 participants
@SimonMarquis @lihenggui