[CPP_CLI] MLC Cli App over JSONEngine interface #3114
Conversation
Comprehensive CPP Cli developed over existing JSONEngine interface. Intended to use in environments with only cli access like Android ADB shells.
|
@mengshyu can you also help to review this pR |
|
Hi @srkreddy1238, It would be good to implement mlc_cli_chat following the structure of the iOS and Android app implementations, which consist of three major components: MLCEngine, JSONFFIEngine, and OpenAIProtocol, as this may enhance consistency and maintainability. The MLCEngine serves as the business logic layer responsible for processing chat requests, managing state, and orchestrating interactions with the inference engine. It ensures efficient handling of streaming responses, allowing smooth and continuous chat interactions. Below this, JSONFFIEngine functions as the execution layer, acting as a bridge between the application logic and the underlying inference engine. It provides an interface for executing model inference, managing background processing, and ensuring efficient communication between the AI model and the rest of the system. At the data layer, OpenAIProtocol defines the standardized structures for handling request and response data, ensuring consistency in how chat interactions are formatted and processed. By maintaining a structured approach to data representation, it enables seamless integration between the business logic and execution layers. Android: https://github.com/mlc-ai/mlc-llm/tree/main/android/mlc4j/src/main/java/ai/mlc/mlcllm iOS: https://github.com/mlc-ai/mlc-llm/tree/main/ios/MLCSwift/Sources/Swift |
|
|
||
| #include "base.h" | ||
|
|
||
| /// Helper function to get the json format of messages |
There was a problem hiding this comment.
You may consider using picojson library
| @@ -158,6 +158,8 @@ if(NOT CARGO_EXECUTABLE) | |||
| message(FATAL_ERROR "Cargo is not found! Please install cargo.") | |||
| endif() | |||
|
|
|||
| add_subdirectory(apps/mlc_cli_chat) | |||
There was a problem hiding this comment.
suggest using USE_CLI_CHAT (default value: false) to determine whether to compile the executable.
|
@mengshyu thanks for the review. We have done this a while back following python chat interface we have. |
Comprehensive CPP Cli developed over existing JSONEngine interface.
Intended to use in environments with only cli access like Android ADB shells.