Set --xla_latency_hiding_scheduler_rerun to 1 #5736
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Summary: This flag will rerun the latency hidding scheduler if the default shared memory limit 95% leads to OOM. Each rerun will choose a value 0.9x of the previous run, and the number of rerun is set to 1 now. Shared memory limit refers to --xla_tpu_scheduler_percent_shared_memory_limit. Lower shared memory limit means less communiation and computation overlapping, and thus worse performance. Test Plan: Tested on Llama 2 7B on V4-32.
|
lgtm, let's wait for the manual run passed then merge this pr |
jonb377
reviewed
Oct 25, 2023
| # Lower shared memory limit means less communiation and computation overlapping, | ||
| # and thus worse performance. | ||
| flags = _set_missing_flags(flags, | ||
| (('xla_latency_hiding_scheduler_rerun', '1'),)) |
There was a problem hiding this comment.
Nice! I was actually wondering about this, we set XLA_FLAGS here, but in other cases we pass the flags through LIBTPU_INIT_ARGS. Do you know if there's a difference? If you saw the appropriate log output from the test, it seems both work...
There was a problem hiding this comment.
oh right.. sorry I missed this. so for everything in compiler/xla/xla.proto we used XLA_FLAGS. xla_latency_hiding_scheduler_rerun is one of those TPU specified flags that needs to be passed in with LIBTPU_INIT_ARGS.
There was a problem hiding this comment.
Good catch. I miss this lol.
There was a problem hiding this comment.
Ah, makes sense that LIBTPU_INIT_ARGS would be tpu-specific lol, thanks @JackCaoG. Is there a rule of thumb to tell which flag goes where? I'm thinking in terms of hashing the compilation environment, I suppose we'll just need to ensure both env vars are included.
|
Thanks for the review, Jon. |
jonb377
pushed a commit
that referenced
this pull request
Oct 31, 2023
Summary: This flag will rerun the latency hidding scheduler if the default shared memory limit 95% leads to OOM. Each rerun will choose a value 0.9x of the previous run, and the number of rerun is set to 1 now. Shared memory limit refers to --xla_tpu_scheduler_percent_shared_memory_limit. Lower shared memory limit means less communiation and computation overlapping, and thus worse performance. Test Plan: Tested on Llama 2 7B on V4-32.
mbzomowski
pushed a commit
to mbzomowski-test-org/xla
that referenced
this pull request
Nov 16, 2023
Summary: This flag will rerun the latency hidding scheduler if the default shared memory limit 95% leads to OOM. Each rerun will choose a value 0.9x of the previous run, and the number of rerun is set to 1 now. Shared memory limit refers to --xla_tpu_scheduler_percent_shared_memory_limit. Lower shared memory limit means less communiation and computation overlapping, and thus worse performance. Test Plan: Tested on Llama 2 7B on V4-32.
ManfeiBai
pushed a commit
that referenced
this pull request
Nov 29, 2023
Summary: This flag will rerun the latency hidding scheduler if the default shared memory limit 95% leads to OOM. Each rerun will choose a value 0.9x of the previous run, and the number of rerun is set to 1 now. Shared memory limit refers to --xla_tpu_scheduler_percent_shared_memory_limit. Lower shared memory limit means less communiation and computation overlapping, and thus worse performance. Test Plan: Tested on Llama 2 7B on V4-32.
ManfeiBai
pushed a commit
that referenced
this pull request
Nov 29, 2023
Summary: This flag will rerun the latency hidding scheduler if the default shared memory limit 95% leads to OOM. Each rerun will choose a value 0.9x of the previous run, and the number of rerun is set to 1 now. Shared memory limit refers to --xla_tpu_scheduler_percent_shared_memory_limit. Lower shared memory limit means less communiation and computation overlapping, and thus worse performance. Test Plan: Tested on Llama 2 7B on V4-32.
chunnienc
pushed a commit
to chunnienc/xla
that referenced
this pull request
Dec 14, 2023
Summary: This flag will rerun the latency hidding scheduler if the default shared memory limit 95% leads to OOM. Each rerun will choose a value 0.9x of the previous run, and the number of rerun is set to 1 now. Shared memory limit refers to --xla_tpu_scheduler_percent_shared_memory_limit. Lower shared memory limit means less communiation and computation overlapping, and thus worse performance. Test Plan: Tested on Llama 2 7B on V4-32.
golechwierowicz
pushed a commit
that referenced
this pull request
Jan 12, 2024
Summary: This flag will rerun the latency hidding scheduler if the default shared memory limit 95% leads to OOM. Each rerun will choose a value 0.9x of the previous run, and the number of rerun is set to 1 now. Shared memory limit refers to --xla_tpu_scheduler_percent_shared_memory_limit. Lower shared memory limit means less communiation and computation overlapping, and thus worse performance. Test Plan: Tested on Llama 2 7B on V4-32.
bhavya01
pushed a commit
that referenced
this pull request
Apr 22, 2024
Summary: This flag will rerun the latency hidding scheduler if the default shared memory limit 95% leads to OOM. Each rerun will choose a value 0.9x of the previous run, and the number of rerun is set to 1 now. Shared memory limit refers to --xla_tpu_scheduler_percent_shared_memory_limit. Lower shared memory limit means less communiation and computation overlapping, and thus worse performance. Test Plan: Tested on Llama 2 7B on V4-32.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary:
This flag will rerun the latency hidding scheduler if the default
shared memory limit 95% leads to OOM. Each rerun will choose a value
0.9x of the previous run, and the number of rerun is set to 1 now.
Shared memory limit refers to --xla_tpu_scheduler_percent_shared_memory_limit.
Lower shared memory limit means less communiation and computation overlapping,
and thus worse performance.
Test Plan:
Tested on Llama 2 7B on V4-32.