Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature] New task plugin Pytorch #11498

Merged
merged 9 commits into from
Aug 18, 2022
Merged

Conversation

jieguangzhou
Copy link
Member

Purpose of the pull request

close: #11497

Brief change log

  • Python environment manager
  • Machine Learning from Git Url
  • Pytorch Example

Verify this pull request

This pull request is code cleanup without any test coverage.

(or)

This pull request is already covered by existing tests, such as (please describe tests).

(or)

This change added tests and can be verified as follows:

(or)

If your pull request contain incompatible change, you should also add it to docs/docs/en/guide/upgrede/incompatible.md

@codecov-commenter
Copy link

codecov-commenter commented Aug 16, 2022

Codecov Report

Merging #11498 (811baff) into dev (2a7e0be) will increase coverage by 0.03%.
The diff coverage is 54.46%.

@@             Coverage Diff              @@
##                dev   #11498      +/-   ##
============================================
+ Coverage     39.36%   39.40%   +0.03%     
- Complexity     4630     4655      +25     
============================================
  Files           987      993       +6     
  Lines         37566    37686     +120     
  Branches       4184     4197      +13     
============================================
+ Hits          14789    14851      +62     
- Misses        21226    21277      +51     
- Partials       1551     1558       +7     
Impacted Files Coverage Δ
...eduler/plugin/task/pytorch/PytorchTaskChannel.java 0.00% <0.00%> (ø)
...plugin/task/pytorch/PytorchTaskChannelFactory.java 0.00% <0.00%> (ø)
...heduler/plugin/task/pytorch/GitProjectManager.java 50.00% <50.00%> (ø)
...phinscheduler/plugin/task/pytorch/PytorchTask.java 54.34% <54.34%> (ø)
...heduler/plugin/task/pytorch/PytorchParameters.java 59.09% <59.09%> (ø)
...cheduler/plugin/task/pytorch/PythonEnvManager.java 86.66% <86.66%> (ø)
...uler/plugin/task/api/k8s/impl/K8sTaskExecutor.java 35.00% <0.00%> (-1.54%) ⬇️
...r/plugin/task/sqoop/parameter/SqoopParameters.java 55.12% <0.00%> (-1.29%) ⬇️
...nscheduler/service/process/ProcessServiceImpl.java 31.87% <0.00%> (ø)
...api/service/impl/ProcessDefinitionServiceImpl.java 32.27% <0.00%> (ø)
... and 3 more

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

support resource list

add pytorch frontend

support git python path

add doc
@jieguangzhou
Copy link
Member Author

PTAL, thanks @SbloodyS @EricGao888 @songjianet @caishunfeng @zhongjiajie

Copy link
Member

@SbloodyS SbloodyS left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's better to use screenshots in english instead of chinese.

@jieguangzhou
Copy link
Member Author

It's better to use screenshots in english instead of chinese.

I used Chinese screenshots in the Chinese document, and the English version of the document used English screenshots.

I am not sure we should use English screenshots both in Chinese document and English version. What do you thinks? @SbloodyS @zhongjiajie

@SbloodyS
Copy link
Member

It's better to use screenshots in english instead of chinese.

I used Chinese screenshots in the Chinese document, and the English version of the document used English screenshots.

I am not sure we should use English screenshots both in Chinese document and English version. What do you thinks? @SbloodyS @zhongjiajie

I vaguely remember that what we asked before was to use English screenshots in both Chinese and English documents. Am I mistaken? @zhongjiajie

@EricGao888
Copy link
Member

It's better to use screenshots in english instead of chinese.

I used Chinese screenshots in the Chinese document, and the English version of the document used English screenshots.
I am not sure we should use English screenshots both in Chinese document and English version. What do you thinks? @SbloodyS @zhongjiajie

I vaguely remember that what we asked before was to use English screenshots in both Chinese and English documents. Am I mistaken? @zhongjiajie

+1, it is unnecessary to maintain two set of screenshots with different language.

@jieguangzhou
Copy link
Member Author

It's better to use screenshots in english instead of chinese.

I used Chinese screenshots in the Chinese document, and the English version of the document used English screenshots.
I am not sure we should use English screenshots both in Chinese document and English version. What do you thinks? @SbloodyS @zhongjiajie

I vaguely remember that what we asked before was to use English screenshots in both Chinese and English documents. Am I mistaken? @zhongjiajie

+1, it is unnecessary to maintain two set of screenshots with different language.

Got it. I will fix it.

jieguangzhou and others added 4 commits August 17, 2022 10:52
fix doc

Co-authored-by: Eric Gao <ericgao.apache@gmail.com>
fix doc

Co-authored-by: Eric Gao <ericgao.apache@gmail.com>
@sonarqubecloud
Copy link

Kudos, SonarCloud Quality Gate passed!    Quality Gate passed

Bug A 0 Bugs
Vulnerability A 0 Vulnerabilities
Security Hotspot A 0 Security Hotspots
Code Smell A 2 Code Smells

62.0% 62.0% Coverage
0.0% 0.0% Duplication

Copy link
Member

@labbomb labbomb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@zhongjiajie zhongjiajie merged commit a974ba7 into apache:dev Aug 18, 2022
@zhongjiajie zhongjiajie changed the title [Feature][Task Plugin] Add Pytorch task plugin to make running machine learning projects easier #11497 [Feature] New task plugin Pytorch Aug 18, 2022
zhuxt2015 pushed a commit to zhuxt2015/dolphinscheduler that referenced this pull request Aug 21, 2022
Co-authored-by: Eric Gao <ericgao.apache@gmail.com>
xdu-chenrj pushed a commit to xdu-chenrj/dolphinscheduler that referenced this pull request Oct 13, 2022
Co-authored-by: Eric Gao <ericgao.apache@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backend document UI ui and front end related
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Feature][Task Plugin] Add Pytorch task plugin to make running machine learning projects easier
6 participants