Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

HDDS-10496. Fetch dependencies using actual build #6359

Merged
merged 1 commit into from
Mar 15, 2024

Conversation

adoroszlai
Copy link
Contributor

What changes were proposed in this pull request?

maven-dependency-plugin seems to ignore exclusions of transitive dependencies (MDEP-844), resulting in larger than necessary Maven cache (including a 250MB jar for aws-java-sdk-bundle via Ranger).

This PR changes the populate-cache workflow to run a regular build for creating the cache (suggested in MDEP-904). This reduces cache size from 750MB to 360MB.

Shading is skipped to speed up the build. A new property maven.shade.skip is added to only skip actual shading while still building (some) ozonefs modules. Only ozonefs-hadoop2 is included, not the ones for Hadoop 3. The latter are unnecessary, since we already get the same dependencies by using the same version of Hadoop for Ozone elsewhere.

Few plugins are not included in the cache this way, but they do not affect build time significantly.

https://issues.apache.org/jira/browse/HDDS-10496

How was this patch tested?

cache build:
https://github.com/adoroszlai/ozone/actions/runs/8215546994/job/22469127579

regular CI (using cache):
https://github.com/adoroszlai/ozone/actions/runs/8215582668

@adoroszlai adoroszlai added the CI label Mar 9, 2024
@adoroszlai adoroszlai self-assigned this Mar 9, 2024
@adoroszlai adoroszlai requested review from smengcl and sadanand48 March 9, 2024 17:59
@adoroszlai adoroszlai requested a review from sadanand48 March 14, 2024 10:25
Copy link
Contributor

@sadanand48 sadanand48 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @adoroszlai for the improvement, LGTM.

@adoroszlai adoroszlai requested a review from errose28 March 14, 2024 11:35
@adoroszlai adoroszlai merged commit b82a2a2 into apache:master Mar 15, 2024
35 checks passed
@adoroszlai adoroszlai deleted the HDDS-10496 branch March 15, 2024 09:34
@adoroszlai
Copy link
Contributor Author

Thanks @sadanand48 for the review.

myskov pushed a commit to myskov/ozone that referenced this pull request Apr 3, 2024
myskov pushed a commit to myskov/ozone that referenced this pull request Apr 3, 2024
xichen01 pushed a commit to xichen01/ozone that referenced this pull request Apr 17, 2024
xichen01 pushed a commit to xichen01/ozone that referenced this pull request Apr 23, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants