[GOBBLIN-1785] add MR_JARS_BASE_DIR and logic to delete old mr jar dirs #3642
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Codecov Report
@@ Coverage Diff @@
## master #3642 +/- ##
=========================================
Coverage 46.86% 46.87%
- Complexity 10758 10759 +1
=========================================
Files 2138 2138
Lines 84007 84018 +11
Branches 9335 9337 +2
=========================================
+ Hits 39368 39381 +13
+ Misses 41049 41042 -7
- Partials 3590 3595 +5
... and 6 files with indirect coverage changes 📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more |
arjun4084346
changed the title
Will-Lo
requested changes
Apr 17, 2023
Comment on lines
681
to
694
| @Deprecated // Deprecated; use MR_JARS_BASE_DIR instead | ||
| public static final String MR_JARS_DIR = "mr.jars.dir"; | ||
| public static final String MR_JARS_BASE_DIR = "mr.jars.base.dir"; |
There was a problem hiding this comment.
Can you include here what is the benefit of using this new configuration key over the old configuration?
| String monthSuffix = new SimpleDateFormat("yyyy-MM").format(System.currentTimeMillis()); | ||
| cleanUpOldJarsDirIfRequired(this.fs, jarsBaseDir); | ||
| this.jarsDir = new Path(jarsBaseDir, monthSuffix); | ||
| } else { |
There was a problem hiding this comment.
I think this would be easier having an else-if,
} else if (this.jobProps.containsKey(ConfigurationKeys.MR_JARS_DIR)) {
this.jarsDir = this.jobProps.getProperty(ConfigurationKeys.MR_JARS_DIR);
} else {
this.jarsDir = this.unsharedJarsDir;
}
There was a problem hiding this comment.
That looks better, just wanted to make minimal changes to show in a diff to the reviewer :D
Will-Lo
approved these changes
Apr 17, 2023
phet
added a commit
to phet/gobblin
that referenced
this pull request
Aug 15, 2023
* upstream/master: [GOBBLIN-1832] Emit warning instead of failing job for retention of Hive Table Views (apache#3695) [GOBBLIN-1831] Use flowexecutionid in kafka monitor and jobnames (apache#3694) [GOBBLIN-1824]Improving the Efficiency of Work Planning in Manifest-Based DistCp Jobs (apache#3686) [GOBBLIN-1829] Fixes bug where the wrong workunit event was being tracked for keepin… (apache#3691) [GOBBLIN-1828] Implement Timeout for Creating Writer Functionality (apache#3690) [GOBBLIN-1827] Add check that if nested field is optional and has a non-null default… (apache#3689) [GOBBLIN-1826] Change isAssignableFrom() to isSuperTypeOf() per Guava 20 javadocs to… (apache#3688) [GOBBLIN-1822]Logging Abnormal Helix Task States (apache#3685) [GOBBLIN-1819] Log helix workflow information and timeout information during submission wait / polling (apache#3681) [GOBBLIN-1821] Let flow execution ID propagate to the Job ID if it exists (apache#3684) [GOBBLIN-1810] Support general iceberg catalog (support configurable behavior for metadata retention policy) (apache#3680) Add null default value to observability events that are additionally added (apache#3682) [GOBBLIN-1816] Add job properties and GaaS instance ID to observability event (apache#3676) [GOBBLIN-1785] add MR_JARS_BASE_DIR and logic to delete old mr jar dirs (apache#3642) initiliaze yarn clients in yarn app launcher so that a child class can override the yarn client creation logic (apache#3679) [GOBBLIN-1811]Fix Iceberg Registration Serialization (apache#3673) [GOBBLIN-1817] change some deprecated code and fix minor codestyle (apache#3678) [GOBBLIN-1812] Mockito should only be test compile (apache#3674) [GOBBLIN-1813] Helix workflows submission timeouts are configurable (apache#3677) [GOBBLIN-1810] Support general iceberg catalog in icebergMetadataWriter (apache#3672) Refactor yarn app launchers to support extending these classes (apache#3671) [GOBBLIN-1808] Bump Guava version from 15.0 to 20.0 (apache#3669) [GOBBLIN-1806] Submit dataset summary event post commit and integrate them into GaaSObservabilityEvent (apache#3667) [GOBBLIN-1814] Add `MRJobLauncher` configurability for any failing mapper to be fatal to the MR job (apache#3675) Add new lookback version finder for use with iceberg retention (apache#3670)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Dear Gobblin maintainers,
Please accept this PR. I understand that it will not be reviewed until I have checked off all the steps below!
JIRA
Description
add a month partitioned jars dir for mr jobs and set up a simple retention policy over it.
more details in https://docs.google.com/document/d/16cHbeHeCpAP79AqE4odxiXs6tLw7wJpCZhgMBSpL3eg/edit
Tests
Commits