running it on mac

[milsun]

how can i run it on my mac m2, llama.cpp doesnt support this as yet?

[zqhuang211]

Currently, inference works on Apple Silicon (cpu or mps), though it’s significantly slower compared to Nvidia GPUs.

If you’d like to try out the Gradio demo on a Mac, simply run just gradio. By default, this will use mps for inference.

Training with --device=mps still need some work, see #91

[zqhuang211]

Ultravox doesn’t support llama.cpp yet. We plan to explore this in the future and definitely welcome contributions from the community.

[milsun]

running it with "mps" would be fp32 I guess, so would need a ton of RAM, right, like ~32GB or so?

[zqhuang211]

Yes for float32. You can also set --data_type=float16, which reduces RAM usage to around 20GB. We haven’t tried quantization yet.

[juberti]

Fixed in #95