Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Feature-10273][Datasource] Add dataSource for AWS Athena #11157

Merged
merged 5 commits into from
Sep 16, 2022
Merged

[Feature-10273][Datasource] Add dataSource for AWS Athena #11157

merged 5 commits into from
Sep 16, 2022

Conversation

guodongym
Copy link
Contributor

Purpose of the pull request

add dataSource for AWS Athena

close #10273

Brief change log

  • Add the ATHENA backend
  • Add the ATHENA test case
  • Add the ATHENA frontend

Verify this pull request

This pull request is code cleanup without any test coverage.

(or)

This pull request is already covered by existing tests, such as (please describe tests).

(or)

This change added tests and can be verified as follows:

@github-actions github-actions bot added backend UI ui and front end related labels Jul 26, 2022
@codecov-commenter
Copy link

codecov-commenter commented Jul 26, 2022 •
edited
Loading

Codecov Report

Merging #11157 (661209f) into dev (764a243) will decrease coverage by 1.81%.
The diff coverage is 59.09%.

@@             Coverage Diff              @@
##                dev   #11157      +/-   ##
============================================
- Coverage     40.39%   38.58%   -1.82%     
+ Complexity     4877     4058     -819     
============================================
  Files           950     1001      +51     
  Lines         37186    36770     -416     
  Branches       4079     4273     +194     
============================================
- Hits          15021    14186     -835     
- Misses        20642    20960     +318     
- Partials       1523     1624     +101
Impacted Files Coverage Δ
...ce/api/datasource/AbstractDataSourceProcessor.java 17.39% <0.00%> (ø)
...gin/datasource/athena/AthenaDataSourceChannel.java 0.00% <0.00%> (ø)
...asource/athena/AthenaDataSourceChannelFactory.java 0.00% <0.00%> (ø)
...ugin/datasource/athena/AthenaDataSourceClient.java 0.00% <0.00%> (ø)
...asource/athena/param/AthenaDataSourceParamDTO.java 0.00% <0.00%> (ø)
.../org/apache/dolphinscheduler/spi/enums/DbType.java 0.00% <0.00%> (ø)
...g/apache/dolphinscheduler/spi/utils/Constants.java 0.00% <ø> (ø)
...source/athena/param/AthenaDataSourceProcessor.java 70.00% <70.00%> (ø)
...datasource/athena/param/AthenaConnectionParam.java 80.00% <80.00%> (ø)
...ler/server/worker/metrics/WorkerServerMetrics.java 0.00% <0.00%> (-72.92%) ⬇️
... and 439 more

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

@SbloodyS SbloodyS added this to the 3.1.0-alpha milestone Jul 27, 2022
@EricGao888 EricGao888 added the miss:docs missing documents in PR label Aug 3, 2022
EricGao888
EricGao888 requested changes Aug 3, 2022
View reviewed changes
Copy link
Member

@EricGao888 EricGao888 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @guodongym , Thx for contributing this great feature! It seems there is no related docs for this PR, could you please add the docs? Thx again.

@guodongym
Copy link
Contributor Author

Hi @guodongym , Thx for contributing this great feature! It seems there is no related docs for this PR, could you please add the docs? Thx again.

Yes I can

@guodongym
Copy link
Contributor Author

Hi @guodongym , Thx for contributing this great feature! It seems there is no related docs for this PR, could you please add the docs? Thx again.

The document has been added

@guodongym
[Feature-10273][Datasource] Add dataSource for AWS Athena
03eedab 
* Add the ATHENA backend
* Add the ATHENA test case
* Add the ATHENA docs
@sonarcloud
Copy link

sonarcloud bot commented Aug 8, 2022

SonarCloud Quality Gate failed.    Quality Gate failed

Bug C 21 Bugs
Vulnerability E 9 Vulnerabilities
Security Hotspot E 37 Security Hotspots
Code Smell A 2101 Code Smells

37.9% 37.9% Coverage
2.4% 2.4% Duplication

@caishunfeng
Copy link
Contributor

SonarCloud Quality Gate failed.    Quality Gate failed

Bug C 21 Bugs Vulnerability E 9 Vulnerabilities Security Hotspot E 37 Security Hotspots Code Smell A 2101 Code Smells

37.9% 37.9% Coverage 2.4% 2.4% Duplication

I found the sonar result is wrong, seems checked all the code, how can we retry this check again?

@guodongym guodongym closed this Aug 12, 2022
@guodongym guodongym reopened this Aug 12, 2022
@guodongym
Copy link
Contributor Author

SonarCloud Quality Gate failed.    Quality Gate failed
Bug C 21 Bugs Vulnerability E 9 Vulnerabilities Security Hotspot E 37 Security Hotspots Code Smell A 2101 Code Smells
37.9% 37.9% Coverage 2.4% 2.4% Duplication

I found the sonar result is wrong, seems checked all the code, how can we retry this check again?

Done

caishunfeng
caishunfeng previously approved these changes Aug 13, 2022
View reviewed changes
Copy link
Contributor

@caishunfeng caishunfeng left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM of the backend part.

@zhongjiajie
Copy link
Member

@songjianet @Amy0104 @devosend do you have time to take a look the FE part?

devosend
devosend previously approved these changes Aug 29, 2022
View reviewed changes
Copy link
Contributor

@devosend devosend left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM for front end

@zhongjiajie zhongjiajie dismissed stale reviews from devosend and caishunfeng via 8def3b2 August 29, 2022 07:48
@zhongjiajie
Copy link
Member

I add missing doc side bar in 8def3b2

@zhongjiajie
Copy link
Member

And will merge PR when it pass

@zhongjiajie
Copy link
Member

Restart the failed test

@zhongjiajie zhongjiajie self-requested a review August 29, 2022 15:08
EricGao888
EricGao888 previously approved these changes Sep 8, 2022
View reviewed changes
Copy link
Member

@EricGao888 EricGao888 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Backend part LGTM except few nits.

...ava/org/apache/dolphinscheduler/plugin/datasource/athena/param/AthenaDataSourceParamDTO.java Show resolved Hide resolved
...rg/apache/dolphinscheduler/plugin/datasource/athena/param/AthenaDataSourceProcessorTest.java Show resolved Hide resolved
@guodongym
[Feature-10273][Datasource] Add dataSource for AWS Athena
661209f 
* use lombok @DaTa here to help the data class look neat.
@sonarcloud
Copy link

sonarcloud bot commented Sep 14, 2022

SonarCloud Quality Gate failed.    Quality Gate failed

Bug C 5 Bugs
Vulnerability A 0 Vulnerabilities
Security Hotspot A 0 Security Hotspots
Code Smell A 29 Code Smells

44.9% 44.9% Coverage
3.2% 3.2% Duplication

zhongjiajie
zhongjiajie approved these changes Sep 15, 2022
View reviewed changes
Copy link
Member

@zhongjiajie zhongjiajie left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, do you have further suggestions? @EricGao888

@kezhenxu94 kezhenxu94 merged commit 86291bb into apache:dev Sep 16, 2022
xdu-chenrj pushed a commit to xdu-chenrj/dolphinscheduler that referenced this pull request Sep 17, 2022
@guodongym @xdu-chenrj
[Feature-10273][Datasource] Add dataSource for AWS Athena (apache#11157)
e4c4806
caishunfeng pushed a commit to caishunfeng/dolphinscheduler that referenced this pull request Sep 19, 2022
@guodongym @caishunfeng
[Feature-10273][Datasource] Add dataSource for AWS Athena (apache#11157)
8c4074c
@caishunfeng caishunfeng mentioned this pull request Sep 19, 2022
Merged
caishunfeng added a commit that referenced this pull request Sep 19, 2022
@caishunfeng @guodongym
[Feature-10273][Datasource] Add dataSource for AWS Athena (#11157) (#…
113dc4b 
…12039)

Co-authored-by: ZhaoGuodong <guodongym@163.com>
@caishunfeng caishunfeng added the release cherry-pick Mark this issue/PR had cherry-pick for release version label Sep 19, 2022
xdu-chenrj pushed a commit to xdu-chenrj/dolphinscheduler that referenced this pull request Oct 13, 2022
@guodongym @xdu-chenrj
[Feature-10273][Datasource] Add dataSource for AWS Athena (apache#11157)
09dece2
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@zhongjiajie zhongjiajie zhongjiajie approved these changes

@EricGao888 EricGao888 EricGao888 left review comments

@caishunfeng caishunfeng caishunfeng left review comments

@devosend devosend devosend left review comments

@kezhenxu94 kezhenxu94 Awaiting requested review from kezhenxu94

@songjianet songjianet Awaiting requested review from songjianet songjianet is a code owner

@Amy0104 Amy0104 Awaiting requested review from Amy0104

@Tianqi-Dotes Tianqi-Dotes Awaiting requested review from Tianqi-Dotes

Assignees

@guodongym guodongym

Labels
backend document miss:docs missing documents in PR release cherry-pick Mark this issue/PR had cherry-pick for release version UI ui and front end related
Projects
None yet
Milestone
3.1.0
Development

Successfully merging this pull request may close these issues.

[Feature][Datasource] Add dataSource for AWS Athena
8 participants
@guodongym @codecov-commenter @caishunfeng @zhongjiajie @devosend @EricGao888 @kezhenxu94 @SbloodyS