v0.3.0

Highlights

• New model topology feature: ISQ and device mapping
• 🔥Faster FlashAttention support when batching
• Removed plotly and associated JS dependencies
• φ³ Support Phi 3.5, Phi 3.5 vision, Phi 3.5 MoE
• Improved Rust API ergonomics
• Support multiple (shaded) GGUF files

MSRV

The Rust MSRV of this version is 1.79.0

What's Changed

• Fixes for auto dtype selection with RUST_BACKTRACE=1 by @EricLBuehler in #690
• Add support multiple GGUF files by @EricLBuehler in #692
• Refactor normal and vision loaders by @EricLBuehler in #693
• Fix split.count GGUF duplication handling by @EricLBuehler in #695
• Batching example by @EricLBuehler in #694
• Some fixes by @EricLBuehler in #697
• Improve vision rust examples by @EricLBuehler in #698
• Add ISQ topology by @EricLBuehler in #701
• Add custom logits processor API by @EricLBuehler in #702
• Add Gemma 2 PagedAttention support by @EricLBuehler in #704
• Faster RmsNorm in Gemma/Gemma2 by @EricLBuehler in #703
• Fix bug in Metal ISQ by @EricLBuehler in #706
• Support GGUF BF16 tensors by @EricLBuehler in #691
• Better support for FlashAttention: real batching + sliding window + softcap by @EricLBuehler in #707
• Remove some usages of pub in models by @EricLBuehler in #708
• Support the Phi 3.5 V model by @EricLBuehler in #710
• Implement the Phi 3.5 MoE model by @EricLBuehler in #709
• Device map topology by @EricLBuehler in #717
• Implement DRY penalty by @EricLBuehler in #637
• Remove plotly and just output CSV loss file by @EricLBuehler in #700
• Using once_cell to reduce MSRV by @EricLBuehler in #724
• Fixes for Windows build by @EricLBuehler in #729
• Even more phi3.5moe fix attempts by @EricLBuehler in #731
• Add example for Phi 3.5 MoE by @EricLBuehler in #733
• Add Phi 3.5 chat template by @EricLBuehler in #734
• Patch ISQ for Mixtral by @EricLBuehler in #730
• Gracefully handle Engine Drop with termination request by @EricLBuehler in #735
• feat(vision): add support for proper file and data image URLs by @Schuwi in #727
• Add new parsing to Python API by @EricLBuehler in #737
• Remove test and add custom error type to Python API by @EricLBuehler in #738
• Update kernels for metal bf16 by @EricLBuehler in #719
• Better Response Result API by @EricLBuehler in #739
• More Metal quantized kernel fixes by @EricLBuehler in #740
• [Breaking] Bump version to v0.3.0 by @EricLBuehler in #736
• Final changes for v0.3.0 by @EricLBuehler in #741

New Contributors

• @Schuwi made their first contribution in #727

Full Changelog: v0.2.5...v0.3.0

v0.2.5

What's Changed

• Refactor ISQ quant parsing by @EricLBuehler in #664
• Refactor server examples to use OpenAI Python client by @EricLBuehler in #665
• Implement prompt chunking by @EricLBuehler in #623
• Python example and server example cleanup by @EricLBuehler in #668
• Implement GPTQ quantization by @EricLBuehler in #467
• Update deps by @EricLBuehler in #672
• Rework the automatic dtype selection feature by @EricLBuehler in #676
• Fix backend Candle fork Metal, flash attn, also Llama linear by @EricLBuehler in #681
• Use converted tokenizer.json in tests by @EricLBuehler in #682
• Refactor ISQ and mistralrs-quant by @EricLBuehler in #683
• Fix metal build for isq by @EricLBuehler in #686
• Add missing error case in automatic dtype selection feature by @ac3xx in #685
• fix null in tool type response by @wseaton in #687
• Implement HQQ quantization by @EricLBuehler in #677
• Bump version to 0.2.5 by @EricLBuehler in #688

New Contributors

• @ac3xx made their first contribution in #685
• @wseaton made their first contribution in #687

Full Changelog: v0.2.4...v0.2.5

Install mistralrs-server 0.2.5

Install prebuilt binaries via shell script

curl --proto '=https' --tlsv1.2 -LsSf https://github.com/EricLBuehler/mistral.rs/releases/download/v0.2.5/mistralrs-server-installer.sh | sh

Download mistralrs-server 0.2.5

File	Platform	Checksum
mistralrs-server-aarch64-apple-darwin.tar.xz	Apple Silicon macOS	checksum
mistralrs-server-x86_64-apple-darwin.tar.xz	Intel macOS	checksum
mistralrs-server-x86_64-unknown-linux-gnu.tar.xz	x64 Linux	checksum

v0.2.4

What's Changed

• fix build on metal by returning Device by @rgbkrk in #642
• Add invite to Matrix chatroom by @EricLBuehler in #644
• Make sure we don't have dead links by @EricLBuehler in #647
• Fix more links by @EricLBuehler in #648
• Throughput for interactive mode by @EricLBuehler in #655
• Implement tool calling by @EricLBuehler in #649
• Fix device map check for paged attn by @EricLBuehler in #656
• Fix for mistral nemo in gguf by @EricLBuehler in #657
• Fix check of cache config when device mapping + PA by @EricLBuehler in #658
• Biollama in tool calling example by @EricLBuehler in #659
• Biollama in tool calling example by @EricLBuehler in #660
• Examples for simple tool calling by @EricLBuehler in #661
• Bump version to 0.2.4 by @EricLBuehler in #662

New Contributors

• @rgbkrk made their first contribution in #642

Full Changelog: v0.2.3...v0.2.4

MSRV

MSRV is 1.75

Install mistralrs-server 0.2.4

Install prebuilt binaries via shell script

curl --proto '=https' --tlsv1.2 -LsSf https://github.com/EricLBuehler/mistral.rs/releases/download/v0.2.4/mistralrs-server-installer.sh | sh

Download mistralrs-server 0.2.4

File	Platform	Checksum
mistralrs-server-aarch64-apple-darwin.tar.xz	Apple Silicon macOS	checksum
mistralrs-server-x86_64-apple-darwin.tar.xz	Intel macOS	checksum
mistralrs-server-x86_64-unknown-linux-gnu.tar.xz	x64 Linux	checksum

v0.2.3

What's Changed

• Implement min-p sampling by @EricLBuehler in #625
• Tweak handling when PA cannot allocate by @EricLBuehler in #632
• Update deps by @EricLBuehler in #633
• Improve penalty context window calculation by @EricLBuehler in #636
• Allow setting PagedAttention KV cache allocation from context size by @EricLBuehler in #640
• Bump version to 0.2.3 by @EricLBuehler in #638

Full Changelog: v0.2.2...v0.2.3

Install mistralrs-server 0.2.3

Install prebuilt binaries via shell script

curl --proto '=https' --tlsv1.2 -LsSf https://github.com/EricLBuehler/mistral.rs/releases/download/v0.2.3/mistralrs-server-installer.sh | sh

Download mistralrs-server 0.2.3

File	Platform	Checksum
mistralrs-server-aarch64-apple-darwin.tar.xz	Apple Silicon macOS	checksum
mistralrs-server-x86_64-apple-darwin.tar.xz	Intel macOS	checksum
mistralrs-server-x86_64-unknown-linux-gnu.tar.xz	x64 Linux	checksum

v0.2.2

What's Changed

• Fix ctrlc handling for scheduler v2 by @EricLBuehler in #614
• Make sliding_window optional for mixtral by @csicar in #616
• Support Llama 3.1 scaled rope by @EricLBuehler in #618

New Contributors

• @csicar made their first contribution in #616

Full Changelog: v0.2.1...v0.2.2

MSRV

MSRV is 1.75.

Install mistralrs-server 0.2.2

Install prebuilt binaries via shell script

curl --proto '=https' --tlsv1.2 -LsSf https://github.com/EricLBuehler/mistral.rs/releases/download/v0.2.2/mistralrs-server-installer.sh | sh

Download mistralrs-server 0.2.2

File	Platform	Checksum
mistralrs-server-aarch64-apple-darwin.tar.xz	Apple Silicon macOS	checksum
mistralrs-server-x86_64-apple-darwin.tar.xz	Intel macOS	checksum
mistralrs-server-x86_64-unknown-linux-gnu.tar.xz	x64 Linux	checksum

v0.2.1

What's Changed

• Fix path normalize for mistralrs-paged-attn by @EricLBuehler in #592
• ISQ python example by @EricLBuehler in #593
• Add support for mistral nemo by @EricLBuehler in #595
• Fix dtype with QLinear by @EricLBuehler in #600
• Update paged-attn build.rs with NVCC flags by @joshpopelka20 in #604
• Bump openssl from 0.10.64 to 0.10.66 by @dependabot in #605
• Update GitHub issue templates by @EricLBuehler in #607
• Add server throughput logging by @EricLBuehler in #608
• Make the plotly feature optional by @EricLBuehler in #597
• Use OnceLock for Python bindings device by @EricLBuehler in #602
• Topk for X-LoRA scalings by @EricLBuehler in #609
• Fix server cross-origin errors by @openmynet in #610
• Refactor sampler by @EricLBuehler in #611
• Bump version to 0.2.1 by @EricLBuehler in #613

New Contributors

• @dependabot made their first contribution in #605
• @openmynet made their first contribution in #610

Full Changelog: v0.2.0...v0.2.1

Install mistralrs-server 0.2.1

Install prebuilt binaries via shell script

curl --proto '=https' --tlsv1.2 -LsSf https://github.com/EricLBuehler/mistral.rs/releases/download/v0.2.1/mistralrs-server-installer.sh | sh

Download mistralrs-server 0.2.1

File	Platform	Checksum
mistralrs-server-aarch64-apple-darwin.tar.xz	Apple Silicon macOS	checksum
mistralrs-server-x86_64-apple-darwin.tar.xz	Intel macOS	checksum
mistralrs-server-x86_64-unknown-linux-gnu.tar.xz	x64 Linux	checksum

v0.2.0

New features

• Support .bin, .pt, .pth extensions
• Add Starcoder 2 GGUF
• 🔥 PagedAttention - beating llama.cpp running GGUF plus all the throughput benefits 😉
• Optimized performance and memory usage

Rust MSRV

MSRV of mistral.rs v0.2.0 is 1.75.

What's Changed

• Fix SWA order (flip it) for Gemma 2 by @EricLBuehler in #554
• Support .bin, .pt, .pth extensions by @EricLBuehler in #557
• Update readme by @EricLBuehler in #558
• Fix Starcoder 2 ISQ by @EricLBuehler in #559
• Update deps by @EricLBuehler in #560
• Add the starcoder2 GGUF arch by @EricLBuehler in #561
• Readme update for starcoder2 gguf by @EricLBuehler in #562
• Fix PyPI release trigger by @EricLBuehler in #566
• Optimize multi-batch and inference performance with PagedAttention by @EricLBuehler in #552
• [Breaking] Version 0.2.0 by @EricLBuehler in #527
• Paged attention support for vision models by @EricLBuehler in #567
• Automatically use paged attn on cuda, get memory size by @EricLBuehler in #568
• Add docs link for vision loader by @EricLBuehler in #570
• Add matching for valid model weight names by @EricLBuehler in #571
• Remove ensure about no paged attn for vision models by @EricLBuehler in #573
• Add percentage utilization support to paged attn by @EricLBuehler in #574
• Include block engine in paged attn metadata by @EricLBuehler in #576
• Update deps and sync Candle by @EricLBuehler in #578
• Optimize CLIP model by @EricLBuehler in #579
• Use softmax_last_dim in CLIP by @EricLBuehler in #580
• Fix method of calculating paged attn with util percent by @EricLBuehler in #581
• Handle windows in paged attn build by @EricLBuehler in #577
• Warn instead of error when paged attn not supported by @EricLBuehler in #583
• Warn instead of error when paged attn for adapters not supported by @EricLBuehler in #584
• Add support for lm_head to adapter models by @EricLBuehler in #586
• Add default plotly feature by @EricLBuehler in #587
• Improve memory handling of PagedAttention with GGUF by @EricLBuehler in #590
• Fix Windows build on cuda w/ PagedAttention by @EricLBuehler in #589
• Update cuda kernels build.rs on windows by @EricLBuehler in #591
• Bump version to 0.2.0 and update docs by @EricLBuehler in #582

Full Changelog: v0.1.26...v0.2.0

Install mistralrs-server 0.2.0

Install prebuilt binaries via shell script

curl --proto '=https' --tlsv1.2 -LsSf https://github.com/EricLBuehler/mistral.rs/releases/download/v0.2.0/mistralrs-server-installer.sh | sh

Download mistralrs-server 0.2.0

File	Platform	Checksum
mistralrs-server-aarch64-apple-darwin.tar.xz	Apple Silicon macOS	checksum
mistralrs-server-x86_64-apple-darwin.tar.xz	Intel macOS	checksum
mistralrs-server-x86_64-unknown-linux-gnu.tar.xz	x64 Linux	checksum

v0.1.26

What's Changed

• Reference cargo dist in readme by @EricLBuehler in #536
• Fix xlora table in readme by @EricLBuehler in #537
• Warning about device mapping for anymoe by @EricLBuehler in #538
• Add updater and installer programs to cargo dist functionality by @EricLBuehler in #540
• Device mapping for AnyMoE by @EricLBuehler in #539
• Add tracking of memory usage by @EricLBuehler in #392
• Fix memory avail for cuda by @EricLBuehler in #543
• Add LLaVA Support by @chenwanqq in #484
• Device mapping fix for normal llama by @EricLBuehler in #549
• Add benchmark for a6000 by @EricLBuehler in #550
• Bump version to 0.1.26 by @EricLBuehler in #551

Full Changelog: v0.1.25...v0.1.26

Install mistralrs-server 0.1.26

Install prebuilt binaries via shell script

curl --proto '=https' --tlsv1.2 -LsSf https://github.com/EricLBuehler/mistral.rs/releases/download/v0.1.26/mistralrs-server-installer.sh | sh

Download mistralrs-server 0.1.26

File	Platform	Checksum
mistralrs-server-aarch64-apple-darwin.tar.xz	Apple Silicon macOS	checksum
mistralrs-server-x86_64-apple-darwin.tar.xz	Intel macOS	checksum
mistralrs-server-x86_64-unknown-linux-gnu.tar.xz	x64 Linux	checksum

v0.1.25

Summary

• Added AnyMoE
• Added the Starcoder 2 model
• Vision interactive mode
• X-LoRA and LoRA support for Gemma 2
• Optimized local loading speed
• Release using cargo dist

MSRV

cargo msrv

MSRV of mistral.rs is 1.75.0.

What's Changed

• Add a vision interactive mode by @EricLBuehler in #468
• Add support for X-LoRA/LoRA to Gemma 2 by @EricLBuehler in #499
• Add adapter activation support for Gemma 2 LoRA by @EricLBuehler in #500
• Decrease MSRV to 1.75.0 by @EricLBuehler in #501
• Fix docs with preprocessor config by @EricLBuehler in #502
• Allow for optional model field in OpenAI requests by @EricLBuehler in #504
• Support streaming for completion by @EricLBuehler in #508
• Select the best device in examples by @EricLBuehler in #507
• Handle Metal error from bf16 autodtype selection by @EricLBuehler in #509
• Use chat messages for the rust examples and show T/s by @EricLBuehler in #510
• Avoid f64 to f32 cast in phirope for dtype on metal by @EricLBuehler in #512
• Handle errors in rust examples by @EricLBuehler in #513
• AnyMoE: Build an MoE model from anything, quickly by @EricLBuehler in #476
• chore: update paths.rs by @eltociear in #516
• Add AnyMoE support for vision models by @EricLBuehler in #515
• Support images in AnyMoE dataset by @EricLBuehler in #517
• Handle metal case for nonzero and bitwise ops by @EricLBuehler in #518
• Support saving and loading AnyMoE gating layer by @EricLBuehler in #519
• Implement the Starcoder 2 model architecture by @EricLBuehler in #522
• Docs for Starcoder 2 by @EricLBuehler in #524
• Starcoder2 docs by @EricLBuehler in #525
• Export loader for starcoder2 by @EricLBuehler in #526
• Support the new phi3 models by @EricLBuehler in #530
• Source the AnyMoE dataset from a structured JSON file by @EricLBuehler in #531
• Create an AnyMoE loss graph by @EricLBuehler in #532
• Add AnyMoE demo video by @EricLBuehler in #533
• Add judgment for whether model_id is a local path to speed up local model loading by @chenwanqq in #523
• Update docs by @EricLBuehler in #534
• release using cargo-dist by @kranurag7 in #480
• Bump version to 0.1.25 by @EricLBuehler in #535

New Contributors

• @kranurag7 made their first contribution in #480

Full Changelog: v0.1.24...v0.1.25

Download mistralrs-server 0.1.25

File	Platform	Checksum
mistralrs-server-aarch64-apple-darwin.tar.xz	Apple Silicon macOS	checksum
mistralrs-server-x86_64-apple-darwin.tar.xz	Intel macOS	checksum
mistralrs-server-x86_64-unknown-linux-gnu.tar.xz	x64 Linux	checksum

v0.1.24

Patch release, please update

What's Changed

• Bump version to 0.1.24 by @EricLBuehler in #497

Full Changelog: v0.1.23...v0.1.24