Preliminary investigation of deployment with localai

[dominic0df]

User story

1. As a software architect
2. I want to preliminary investigate the deployment of our model with localai
3. So that we get familiar with localai and we can already identify possible problems.

Acceptance criteria

• Key documentation of how to integrate our model with localai is identified and consulted
• Identification of possible problems of the integration of the model into localai
  • Possible problems with the integration of our model into localai are identified
  • Possible problems are summarized
  • Possible problems are communicated to the POs so that they can be tackled with issues in the next sprint
• A suggestion is made of how we should set up a frontend to work with localai (either own implementation or usage of an example chatbot with permissive license that may exist)

Further information

• Our industry partner is currently setting up a cluster in which localai is deployed. So localai will run on the target infrastructure where we should deploy our model and we should make our model accessible using the API of localai
• see https://localai.io/

Definition of done (DoD)

• Bill of Materials in the planning document has been updated
• All feature branches have been merged and closed
• New feature code has been documented
• Potential new licenses have been checked
• All GitHub Actions are passing
• The requirement.txt is updated

DoD general criteria

• Feature has been fully implemented
• Feature has been merged into the mainline
• All acceptance criteria were met
• Product owner approved features
• All tests are passing
• Developers agreed to release

[dnsch]

Report on LocalAI

We will deploy our using localAI on a Kubermatic cluster.

Findings

1. It is entirely possible to deploy localAI on a Kubernetes cluster (see: https://localai.io/basics/kubernetes/). It is unclear however if this also works smoothly with Kubermatic, This will require further testing on the Hetzner machine.
2. It's also possible to use a GUI for interacting with the LLM (see: feat(ux): Add chat, tts, and image-gen pages to the WebUI mudler/LocalAI#2222).
3. It also supports GGUF models ("[...] Runs gguf, transformers, diffusers and many more models architectures", https://github.com/mudler/LocalAI).

Conclusion

It will now be important to implement localAI on a Kubermatic cluster. If this succeeds, the GUI for interacting with the LLM should be easily available.