[core][distributed] add zmq fallback for broadcasting large objects #6183
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
youkaichao
commented
Jul 7, 2024
Comment on lines
+415
to
+425
| if self.n_local_reader > 0: | ||
| if len(serialized_obj) >= self.buffer.max_chunk_bytes: | ||
| with self.acquire_write() as buf: | ||
| buf[0] = 1 # overflow | ||
| self.local_socket.send(serialized_obj) | ||
| else: | ||
| with self.acquire_write() as buf: | ||
| buf[0] = 0 # not overflow | ||
| buf[1:len(serialized_obj) + 1] = serialized_obj | ||
| if self.n_remote_reader > 0: | ||
| self.remote_socket.send(serialized_obj) |
There was a problem hiding this comment.
this is the most critical part, previously, when object size is too large (e.g. large image data), vLLM will error. Now, we will fall back to zqm.
|
This idea sounds good, but I don't have much experience with cross-device communication, so I'll leave the review to someone who is more qualified. |
zhuohan123
approved these changes
Jul 9, 2024
|
@youkaichao |
|
@WoosukKwon thanks for testing! I think it works because I added the dependency in |
adityagoel14
pushed a commit
to adityagoel14/vllm-torchrun-test
that referenced
this pull request
Jul 10, 2024
…-project#6183) [core][distributed] add zmq fallback for broadcasting large objects (vllm-project#6183) (cherry picked from commit da78cae)
dtrifiro
pushed a commit
to opendatahub-io/vllm
that referenced
this pull request
Jul 17, 2024
…-project#6183) [core][distributed] add zmq fallback for broadcasting large objects (vllm-project#6183)
xjpang
pushed a commit
to xjpang/vllm
that referenced
this pull request
Jul 24, 2024
…-project#6183) [core][distributed] add zmq fallback for broadcasting large objects (vllm-project#6183)
Alvant
pushed a commit
to compressa-ai/vllm
that referenced
this pull request
Oct 26, 2024
…-project#6183) [core][distributed] add zmq fallback for broadcasting large objects (vllm-project#6183) Signed-off-by: Alvant <alvasian@yandex.ru>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The input to vision language model contains images, which has variable length and can be quite large.
While the shared memory broadcast introduced in #5399 works fine for LLMs, later we find we often need to adjust the buffer size for vision language models.
Estimating the size upper bound can be difficult. To solve the problem, this PR adds a fallback option using zeromq.
In addition, shared memory broadcast is limited to single node, while zeromq (socket-based) is not. Therefore, we can extend the broadcast to also work for cross-node settings. This PR extends the functionality.
cc @DarkLight1337 @ywang96 for vision language model related.