v0.4.36

Mixtral supports

Supports Mixtral on BentoCloud with vLLM and all required dependencies.

Bento built with openllm now defaults to python 3.11 for this change to work.

Installation

pip install openllm==0.4.36

To upgrade from a previous version, use the following command:

pip install --upgrade openllm==0.4.36

Usage

All available models: openllm models

To start a LLM: python -m openllm start HuggingFaceH4/zephyr-7b-beta

To run OpenLLM within a container environment (requires GPUs): docker run --gpus all -it -P -v $PWD/data:$HOME/.cache/huggingface/ ghcr.io/bentoml/openllm:0.4.36 start HuggingFaceH4/zephyr-7b-beta

Find more information about this release in the CHANGELOG.md

What's Changed

• feat(openai): supports echo by @aarnphm in #760
• fix(openai): logprobs when echo is enabled by @aarnphm in #761
• ci: pre-commit autoupdate [pre-commit.ci] by @pre-commit-ci in #767
• chore(deps): bump docker/metadata-action from 5.2.0 to 5.3.0 by @dependabot in #766
• chore(deps): bump actions/setup-python from 4.7.1 to 5.0.0 by @dependabot in #765
• chore(deps): bump taiki-e/install-action from 2.21.26 to 2.22.0 by @dependabot in #764
• chore(deps): bump aquasecurity/trivy-action from 0.14.0 to 0.16.0 by @dependabot in #763
• chore(deps): bump github/codeql-action from 2.22.8 to 2.22.9 by @dependabot in #762
• feat: mixtral support by @aarnphm in #770

Full Changelog: v0.4.35...v0.4.36