Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[GOBBLIN-1829] Fixes bug where the wrong workunit event was being tracked for keepin… #3691

Merged
merged 2 commits into from
May 3, 2023
Merged

[GOBBLIN-1829] Fixes bug where the wrong workunit event was being tracked for keepin… #3691

merged 2 commits into from
May 3, 2023

Conversation

Will-Lo
Copy link
Contributor

@Will-Lo Will-Lo commented May 2, 2023 •
edited
Loading

…g track of work discovery time

Dear Gobblin maintainers,

Please accept this PR. I understand that it will not be reviewed until I have checked off all the steps below!

JIRA

Description

  • Here are some details about my PR, including screenshots (if applicable):

In Gobblin framework, there are 2 events that look similar but do very different functions:

  1. WorkUnitPreparationTimer - which serializes workunits and compacts them into MultiWorkUnits
  2. WorkUnitCreationTimer - which tracks the time taken to discover the work needed for Gobblin to launch a job.

We want to track the latter in GaaS

Tests

  • My PR adds the following unit tests OR does not need testing for this extremely good reason:

Commits

  • My commits all reference JIRA issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "How to write a good git commit message":
    1. Subject is separated from body by a blank line
    2. Subject is limited to 50 characters
    3. Subject does not end with a period
    4. Subject uses the imperative mood ("add", not "adding")
    5. Body wraps at 72 characters
    6. Body explains "what" and "why", not "how"

ZihanLi58
ZihanLi58 reviewed May 2, 2023
View reviewed changes
Copy link
Contributor

@ZihanLi58 ZihanLi58 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the quick fix, just one small question

...n-service/src/main/java/org/apache/gobblin/service/monitoring/KafkaAvroJobStatusMonitor.java
properties.put(TimingEvent.WORKUNIT_PLAN_START_TIME, properties.getProperty(TimingEvent.METADATA_START_TIME));
properties.put(TimingEvent.WORKUNIT_PLAN_END_TIME, properties.getProperty(TimingEvent.METADATA_END_TIME));
break;
case TimingEvent.LauncherTimings.JOB_START:
case TimingEvent.FlowTimings.FLOW_RUNNING:
case TimingEvent.LauncherTimings.JOB_SUMMARY:
case TimingEvent.LauncherTimings.WORK_UNITS_PREPARATION:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we need this to set job status as running?

Copy link
Contributor Author

@Will-Lo Will-Lo May 3, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess we don't in particular, but it helps store all the specific event metadata in the state store. Since if the event is being tracked, then it would add all the event metadata as part of the properties. Otherwise the switch statement will return null so the event gets completely ignored.

@codecov-commenter
Copy link

codecov-commenter commented May 2, 2023 •
edited
Loading

Codecov Report

Merging #3691 (729352a) into master (e47aef7) will increase coverage by 2.51%.
The diff coverage is n/a.

@@             Coverage Diff              @@
##             master    #3691      +/-   ##
============================================
+ Coverage     46.99%   49.50%   +2.51%     
+ Complexity    10788     9311    -1477     
============================================
  Files          2138     1756     -382     
  Lines         84040    68394   -15646     
  Branches       9340     7794    -1546     
============================================
- Hits          39497    33861    -5636     
+ Misses        40955    31404    -9551     
+ Partials       3588     3129     -459
Impacted Files Coverage Δ
.../service/monitoring/KafkaAvroJobStatusMonitor.java 0.00% <ø> (ø)

... and 391 files with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

ZihanLi58
ZihanLi58 approved these changes May 3, 2023
View reviewed changes
Copy link
Contributor

@ZihanLi58 ZihanLi58 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

@Will-Lo Will-Lo merged commit b5d2558 into apache:master May 3, 2023
phet added a commit to phet/gobblin that referenced this pull request Aug 15, 2023
@phet
Merge remote-tracking branch 'upstream/master'
90dd1ff 
* upstream/master:
  [GOBBLIN-1832] Emit warning instead of failing job for retention of Hive Table Views (apache#3695)
  [GOBBLIN-1831] Use flowexecutionid in kafka monitor and jobnames (apache#3694)
  [GOBBLIN-1824]Improving the Efficiency of Work Planning in Manifest-Based DistCp Jobs (apache#3686)
  [GOBBLIN-1829] Fixes bug where the wrong workunit event was being tracked for keepin… (apache#3691)
  [GOBBLIN-1828] Implement Timeout for Creating Writer Functionality (apache#3690)
  [GOBBLIN-1827] Add check that if nested field is optional and has a non-null default… (apache#3689)
  [GOBBLIN-1826] Change isAssignableFrom() to isSuperTypeOf() per Guava 20 javadocs to… (apache#3688)
  [GOBBLIN-1822]Logging Abnormal Helix Task States (apache#3685)
  [GOBBLIN-1819] Log helix workflow information and timeout information during submission wait / polling (apache#3681)
  [GOBBLIN-1821] Let flow execution ID propagate to the Job ID if it exists (apache#3684)
  [GOBBLIN-1810] Support general iceberg catalog (support configurable behavior for metadata retention policy) (apache#3680)
  Add null default value to observability events that are additionally added (apache#3682)
  [GOBBLIN-1816] Add job properties and GaaS instance ID to observability event (apache#3676)
  [GOBBLIN-1785] add MR_JARS_BASE_DIR and logic to delete old mr jar dirs (apache#3642)
  initiliaze yarn clients in yarn app launcher so that a child class can override the yarn client creation logic (apache#3679)
  [GOBBLIN-1811]Fix Iceberg Registration Serialization (apache#3673)
  [GOBBLIN-1817] change some deprecated code and fix minor codestyle (apache#3678)
  [GOBBLIN-1812] Mockito should only be test compile (apache#3674)
  [GOBBLIN-1813] Helix workflows submission timeouts are configurable (apache#3677)
  [GOBBLIN-1810] Support general iceberg catalog in icebergMetadataWriter (apache#3672)
  Refactor yarn app launchers to support extending these classes (apache#3671)
  [GOBBLIN-1808] Bump Guava version from 15.0 to 20.0 (apache#3669)
  [GOBBLIN-1806] Submit dataset summary event post commit and integrate them into GaaSObservabilityEvent (apache#3667)
  [GOBBLIN-1814] Add `MRJobLauncher` configurability for any failing mapper to be fatal to the MR job (apache#3675)
  Add new lookback version finder for use with iceberg retention (apache#3670)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ZihanLi58 ZihanLi58 ZihanLi58 approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@Will-Lo @codecov-commenter @ZihanLi58