Support beam search & parallel generation #7
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
zhuohan123
approved these changes
Mar 10, 2023
| probs: torch.Tensor, | ||
| p: torch.Tensor, | ||
| ) -> torch.Tensor: | ||
| # TODO(woosuk): Optimize. |
There was a problem hiding this comment.
Maybe it's faster to simply mask out the tokens whose cumulative gradient is smaller than top_p (example code)?
There was a problem hiding this comment.
Thanks for letting me know the code. I feel that implementation would not be remarkably more efficient than ours, because it includes 2 softmax rather than 1.
v1nc3nt27
pushed a commit
to v1nc3nt27/vllm
that referenced
this pull request
Sep 12, 2023
remove not needed files
xiangyuT
added a commit
to xiangyuT/vllm
that referenced
this pull request
Oct 24, 2023
add torch.cuda.empty_cache()
hongxiayang
pushed a commit
to hongxiayang/vllm
that referenced
this pull request
Feb 13, 2024
slyalin
pushed a commit
to slyalin/vllm
that referenced
this pull request
Mar 22, 2024
Updated OpenVINO version in dockerfile
mzusman
added a commit
to mzusman/vllm
that referenced
this pull request
Apr 16, 2024
* Return support for other models apart from jamba * Support n>1 * Revert 2 commits d054737 'Support n>1' b5167cc 'Return support for other models apart from jamba' * TP on input and output * Basic TP impl , working, correctness not working * TP is working * Roll back the verification that everything in the weights fits into the model * Cleanup * Use world size func * clean up * Import * Apply whitespace suggestions from code review * Organize imports * Add comment on the unsqueeze in conv1d * Organize and remove redundant code in forward pass * Remove print * Add comments Co-authored-by: tomeras91 <57313761+tomeras91@users.noreply.github.com> * White spaces * Set as A * better comment --------- Co-authored-by: Mor Zusman <morz@ai21.com> Co-authored-by: tomeras91 <57313761+tomeras91@users.noreply.github.com>
sfc-gh-hazhang
pushed a commit
to sfc-gh-hazhang/vllm
that referenced
this pull request
May 7, 2024
arctic load quantized checkpoint
This was referenced Jul 5, 2024
Closed
zeroorhero
pushed a commit
to zeroorhero/vllm
that referenced
this pull request
Sep 23, 2024
Add support for LMCache
1 task
1 task
This was referenced Oct 12, 2024
Felixllq
pushed a commit
to Felixllq/vllm
that referenced
this pull request
Nov 12, 2024
* llama support * flash_attention * sharded * expend * fix: remove redunctant info * change main * llama and opt model supported --------- Co-authored-by: Shao Siyang FYP PDCL <shaosy@scsehg.cm.cluster> Co-authored-by: lairuiqi <lrq619@outlook.com> Co-authored-by: LaiRuiqi <58351056+lrq619@users.noreply.github.com>
1 task
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR adds support for beam search and parallel generation (i.e.,
n> 1).NOTE: The correctness is only checked for beam search, but not for random sampling methods.
Tested models:
Tested GPUs: