Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

LLMPerfV2 #19

Merged
merged 4 commits into from
Dec 6, 2023
Merged

LLMPerfV2 #19

merged 4 commits into from
Dec 6, 2023

Conversation

avnishn
Copy link
Member

@avnishn avnishn commented Dec 4, 2023

Signed-off-by: Avnish Narayan avnish@anyscale.com

Signed-off-by: Avnish Narayan <avnish@anyscale.com>
Signed-off-by: Avnish Narayan <avnish@anyscale.com>
@xieus xieus requested review from rickyyx and kylehh December 4, 2023 22:27
@xieus xieus added the enhancement New feature or request label Dec 4, 2023
@xieus
Copy link

xieus commented Dec 4, 2023

@avnishn Thanks. As this is a major version upgrade, could you add some description to the PR?

@avnishn
Copy link
Member Author

avnishn commented Dec 4, 2023

The latest version of LLMPerf brings a suite of significant updates designed to provide more in-depth and customizable benchmarking capabilities for LLM inference. These updates include:

  • Expanded metrics with quantile distribution (P25-99): Comprehensive data representation for deeper insights.
  • Customizable benchmarking parameters: Tailor parameters to fit specific use case scenarios.
  • Introduction of load test and correctness test: Assessing performance and accuracy under stress.
  • Broad compatibility: Supports a range of products including Anyscale Endpoints, OpenAI, Anthropic, together.ai, Fireworks.ai, Perplexity, Huggingface, Lepton AI, and various APIs supported by the LiteLLM project).
  • Easy addition of new LLMs via the LLMClient API.

*$py.class

# C extensions
# The build output should clearly not be checked in
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is this file change expected?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah

Signed-off-by: Avnish Narayan <avnish@anyscale.com>
@avnishn avnishn merged commit 1eac866 into main Dec 6, 2023
@avnishn avnishn deleted the llmperfv2 branch December 6, 2023 01:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants