Roadmap 🚧

[numb3r3]

Below you can find the rough roadmap for this product.

v0.1.0

target timeline: June 9, 2023

• prepare a draft README to illustrate the UX of this package we expected
• enable pip installable via the poetry tool and add release runner in CI/CD
• model and tokenizer loading factory implementation based on accelerate and transformers package
• enable device_map='balanced' to allow multi-gpus for large-scale model
• implement jina executor wrapper to serve the model simply based on jina framework
• a command line to allow the user to use the model in one-line command as mentioned in README
• support all of the models included in the MODEL_ZOO.md
• finalize the README and double check all of the code examples works
• implement a gateway to support /generate endpoint to allow simple JSON prompts (now, it only accepts docarray JSON schema)
• integrate with inference-client as the official py-client

v0.2.0

target timeline: June 16, 2023

• support docker container usage
• 8/4-bit quantization integration
• implement server-side-event for streaming style token generation (aka. OpenAI-tyle text generation)
• support checkpoint from LORA finetuning
• support jcloud deployment in one-line command

v0.3.0

target timeline: TBD

• implement an online playground based on gradio package
• support MiniGPT-4 as one more multimodality model
• support post-training quantization via GPTQ approach
• layer-wise distribution deployment (i.e., slice the whole model into several executors). So that the model which cannot be hosted in a single node can be supported with k8s.

Potential features

• allow simple finetune methods
• flexible offload techniques to host model

[numb3r3]

@saahil FYI, this is the overall roadmap