How to run llama3?

[gnumoksha]

Hi. How can I run llama3, with a custom prompt, using LocalAI?

[mudler]

One way would be to can copy this llama3 model YAML in your models folder and start LocalAI: https://github.com/mudler/LocalAI/blob/master/embedded/models/llama3-instruct.yaml. You can edit then the templates in the YAML file as you like.

Alternatively you can point LocalAI to that URL by specifying it as argument:

docker run -p 8080:8080 --name local-ai -ti localai/localai:latest-aio-cpu https://raw.githubusercontent.com/mudler/LocalAI/master/embedded/models/llama3-instruct.yaml

[BradKML]

Side note is that using the default UX (http://localhost:8080/browse/) from LocalAI to download LLaMA3 gets stuck when calling from API with an error that repeats either

• Fails: could not load model: rpc error: code = Unknown desc = failed loading model or
• Fails: could not load model: rpc error: code = Unknown desc = stat /build/models/Meta-Llama-3-8B-Instruct-Q5_K_M.gguf: no such file or directory

Addendum: Seen similar issues with others using AIO with GPU default image #2214 #1946 #1938

P.S. can this CLI operation download the combo of LLM, embedding, and reranker? Cross-reference to #2192