LLM inference - Openai Chat Api or Llama compatible chat params #5558
Labels
stat:awaiting googler
Waiting for Google Engineer's Response
task:LLM inference
Issues related to MediaPipe LLM Inference Gen AI setup
type:feature
Enhancement in the New Functionality or Request for a New Solution
Describe the feature and the current behaviour/state
Does GenAi Llm inference support openai chat api or support taking parameters similar to Llama models? If not then I am requesting implementing the api similar to Openai Chat completion.
Please specify the use cases for this feature
For example having a stop param to stop the generation of the llm inference would be great. And Llama compatible chat format can be useful to integrate with other frameworks such as Langchain.
Any Other info
There is a mention of Llama in a comment in llm.h although I am not sure what this is about.
The text was updated successfully, but these errors were encountered: