Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Streaming openai api support #783

Open
wants to merge 2 commits into
base: main
Choose a base branch
from

Conversation

yunhaoli24
Copy link

Description

Use transformers TextIteratorStreamer to support streaming response for the OpenAI API. #762

Ref https://huggingface.co/docs/transformers/internal/generation_utils

Related Issue

None

@guyuexue
Copy link

choice_data = ChatCompletionResponseStreamChoice(
    index=0, delta=DeltaMessage(content=new_text), finish_reason=None
)
chunk = ChatCompletionResponse(
    model=model_id, choices=[choice_data], object="chat.completion.chunk"
)

代码有些问题,流式接口返回时这块会报错,
ChatCompletionResponseStreamChoiceChatCompletionResponse不匹配,

要在openai_api_protocol.py建一个ChatCompletionResponseStream进行替换

class ChatCompletionResponseStream(BaseModel):
    id: str = Field(default_factory=lambda: f"chatcmpl-{shortuuid.random()}")
    object: str = "chat.completion"
    created: int = Field(default_factory=lambda: int(time.time()))
    model: str = "chinese-llama-alpaca"
    choices: List[ChatCompletionResponseStreamChoice]

@yunhaoli24
Copy link
Author

choice_data = ChatCompletionResponseStreamChoice(
    index=0, delta=DeltaMessage(content=new_text), finish_reason=None
)
chunk = ChatCompletionResponse(
    model=model_id, choices=[choice_data], object="chat.completion.chunk"
)

代码有些问题,流式接口返回时这块会报错, ChatCompletionResponseStreamChoiceChatCompletionResponse不匹配,

要在openai_api_protocol.py建一个ChatCompletionResponseStream进行替换

class ChatCompletionResponseStream(BaseModel):
    id: str = Field(default_factory=lambda: f"chatcmpl-{shortuuid.random()}")
    object: str = "chat.completion"
    created: int = Field(default_factory=lambda: int(time.time()))
    model: str = "chinese-llama-alpaca"
    choices: List[ChatCompletionResponseStreamChoice]

您好,我修改了一下代码,使得ChatCompletionResponse兼容两种格式的返回。现在运行不会报错了

class ChatCompletionResponse(BaseModel):
    id: str = Field(default_factory=lambda: f"chatcmpl-{shortuuid.random()}")
    object: str = "chat.completion"
    created: int = Field(default_factory=lambda: int(time.time()))
    model: str = "chinese-llama-alpaca"
    choices: List[
        Union[ChatCompletionResponseChoice, ChatCompletionResponseStreamChoice]
    ]

@airaria
Copy link
Contributor

airaria commented Aug 1, 2023

Thank you for your contribution to our project.
Since we are working on Chinese-LLaMA-Alpaca-2, and it is still under construction, we will be very glad to accept PR to Chinese-LLaMA-Alpaca-2. Would you please create a PR on Streaming openai api support to Chinese-LLaMA-Alpaca-2? Most of the code of openai_api_protocol.py and openai_api_server.py in the two projects are the same, except for some prompts and naming conventions. We really appreciate for your contribution.

As long as we have reviewed your code, we will merge your PRs in both repos.

@yunhaoli24
Copy link
Author

Thank you for your contribution to our project. Since we are working on Chinese-LLaMA-Alpaca-2, and it is still under construction, we will be very glad to accept PR to Chinese-LLaMA-Alpaca-2. Would you please create a PR on Streaming openai api support to Chinese-LLaMA-Alpaca-2? Most of the code of openai_api_protocol.py and openai_api_server.py in the two projects are the same, except for some prompts and naming conventions. We really appreciate for your contribution.

As long as we have reviewed your code, we will merge your PRs in both repos.

ok

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants