Add replace_with_parallel_cross_entropy flag #9579
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Thanks for your contribution! |
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## develop #9579 +/- ##
===========================================
- Coverage 52.77% 52.76% -0.01%
===========================================
Files 709 710 +1
Lines 111172 111235 +63
===========================================
+ Hits 58672 58695 +23
- Misses 52500 52540 +40 ☔ View full report in Codecov by Sentry. |
liym27
reviewed
Dec 9, 2024
| @@ -74,6 +74,7 @@ python -u -m paddle.distributed.launch \ | |||
| --do_eval \ | |||
| --device "gpu" \ | |||
| --data_impl "mmap" \ | |||
| --replace_with_parallel_cross_entropy 0 \ | |||
There was a problem hiding this comment.
Would it be better to place replace_with_parallel_cross_entropy in tensor_parallel_config configuration?
There was a problem hiding this comment.
Done
Move replacew_ith_marallel_cross_detropy to tensor_marallel_config for configuration.
waliwali777
force-pushed
the
add_replace_cross_entropy_flag
branch
9 times, most recently
from
0db0cef to
e38c9d8
Compare
liym27
reviewed
Dec 16, 2024
scripts/distribute/ci_case_auto.sh
Outdated
| @@ -839,6 +840,103 @@ function llama_pir_auto_fuse_ffn_attention_qkv_MP2() { | |||
| echo "=========== $FUNCNAME run end ===========" | |||
| } | |||
|
|
|||
| function llama_pir_auto_replace_with_parallel_cross_entropy_MP2() { | |||
There was a problem hiding this comment.
这样的写法会增加2个测试。建议直接在已有的测试上，加上 parallel cross entropy 参数验证，只需增加一个测试。
waliwali777
force-pushed
the
add_replace_cross_entropy_flag
branch
from
252a2d0 to
d16239c
Compare
waliwali777
force-pushed
the
add_replace_cross_entropy_flag
branch
from
5a1ee7b to
55eca3e
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
PR types
Others
PR changes
Others
Description
为
replace_with_parallel_cross_entropy pass添加开关，可以在脚本进行开关控制开 pass：
--tensor_parallel_config "replace_with_parallel_cross_entropy"关 pass：
--tensor_parallel_config ""