ADR: Breaking models weights out of model images #752
Conversation
✅ Deploy Preview for leapfrogai-docs canceled.
|
alekst23
added this to the Current - RAG UX Enhancements | Model Directory | API Odds and Ends milestone
| Proposed | ||
|
|
||
| ## Context | ||
| GenerativeAI models are big. Because LeapfrogAI is designed to be deployable into AirGapped environments, we need to ensure that we are bringing the big GenerativeAI models with us. Currently, we are brining the AI models with us by backing them into our container images. For example, [we download synthia into our llama-cpp-python image](https://github.com/defenseunicorns/leapfrogai/blob/d1e42d9296f6e014ffbbcec2ba295443b1675567/packages/llama-cpp-python/Dockerfile#L15) and here we [download whisper](https://github.com/defenseunicorns/leapfrogai/blob/d1e42d9296f6e014ffbbcec2ba295443b1675567/packages/whisper/Dockerfile#L14) into our whisper image. Some of the models we are trying to use are large (several GBs). |
There was a problem hiding this comment.
nit: mispelled "baking"
| - The initialization time of pods is increased because of time spent moving the containers OCI layers into the pod. | ||
|
|
||
| ## Decision | ||
| While no decision has been made yet, I am leaning towards proposing we go with the simplest solution of using PVCs to manage our GenAI models. |
There was a problem hiding this comment.
Since this is still in "proposed" status, I would state PVCs as the decision rather than "leaning towards proposing" ...you are proposing!
|
|
||
|
|
||
| ## Rationale | ||
| N/A as no [Decision](#Decision) has been made yet. |
There was a problem hiding this comment.
Provide (proposed) rationale for PVC path forward
| ### Raw PVC Attachments | ||
| [k8s PVC Docs](https://kubernetes.io/docs/concepts/storage/persistent-volumes/) | ||
|
|
||
| Maybe the simplest solution is the best solution? We can create a PersistantVolume for each model that gets populated during deploy time. This PersistantVolume will be mounted by all of the Pods that want to use that model. |
There was a problem hiding this comment.
You say PersistantVolume here, but then use the acronym PVC (PersistentVolumeClaim) elsewhere, which is a request for the PV resources. Be consistent and/or clarify the relationship in a way non-k8s folks like me can understand easily when reading.
Would also be good to define the PVC acronym somewhere in this doc. You currently spell out PersistantVolume, but jump straight to using PVC as an acronym.
|
|
||
|
|
||
| Cons of PVC: | ||
| - Hard to optimize re-deploys (If the model weights don't change) and benefit from caching. |
There was a problem hiding this comment.
Can you elaborate on why it's hard to optimize re-deploys? (Maybe it's obvious to k8s folks, but I don't understand.)
also nit: the "I" in "If" is capitalized
|
Greetings @YrrepNoj |
Related to #623