Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: support stepfun model #1012

Merged
merged 12 commits into from
Jun 19, 2024
Merged

feat: support stepfun model #1012

merged 12 commits into from
Jun 19, 2024

Conversation

Chi-Kai
Copy link
Contributor

@Chi-Kai Chi-Kai commented May 29, 2024

Ⅰ. Describe what this PR did

Support 阶跃星辰(stepfun) model, api documentation (https://platform.stepfun.com)

Ⅱ. Does this pull request fix one issue?

fixes #958

Ⅲ. Why don't you add test cases (unit test/integration test)?

Ⅳ. Describe how to verify it

envoy.yaml:

# File generated by hgctl. Modify as required.

admin:
  address:
    socket_address:
      protocol: TCP
      address: 0.0.0.0
      port_value: 9901
static_resources:
  listeners:
    - name: listener_0
      address:
        socket_address:
          protocol: TCP
          address: 0.0.0.0
          port_value: 10000
      filter_chains:
        - filters:
            - name: envoy.filters.network.http_connection_manager
              typed_config:
                "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
                scheme_header_transformation:
                  scheme_to_overwrite: https
                stat_prefix: ingress_http
                # Output envoy logs to stdout
                access_log:
                  - name: envoy.access_loggers.stdout
                    typed_config:
                      "@type": type.googleapis.com/envoy.extensions.access_loggers.stream.v3.StdoutAccessLog
                # Modify as required
                route_config:
                  name: local_route
                  virtual_hosts:
                    - name: local_service
                      domains: [ "*" ]
                      routes:
                        - match:
                            prefix: "/"
                          route:
                            cluster: stepfun
                            timeout: 300s
                http_filters:
                  - name: wasmtest
                    typed_config:
                      "@type": type.googleapis.com/udpa.type.v1.TypedStruct
                      type_url: type.googleapis.com/envoy.extensions.filters.http.wasm.v3.Wasm
                      value:
                        config:
                          name: wasmtest
                          vm_config:
                            runtime: envoy.wasm.runtime.v8
                            code:
                              local:
                                filename: /etc/envoy/plugin.wasm
                          configuration:
                            "@type": "type.googleapis.com/google.protobuf.StringValue"
                            value: |
                              {
                                "provider": {
                                  "type": "stepfun",
                                  "domain": "api.stepfun.com",
                                  "apiTokens": [
                                    "api_token"
                                  ]
                                }
                              }
                  - name: envoy.filters.http.router
  clusters:
    - name: httpbin
      connect_timeout: 30s
      type: LOGICAL_DNS
      # Comment out the following line to test on v6 networks
      dns_lookup_family: V4_ONLY
      lb_policy: ROUND_ROBIN
      load_assignment:
        cluster_name: httpbin
        endpoints:
          - lb_endpoints:
              - endpoint:
                  address:
                    socket_address:
                      address: httpbin
                      port_value: 80
    - name: stepfun
      connect_timeout: 30s
      type: LOGICAL_DNS
      dns_lookup_family: V4_ONLY
      lb_policy: ROUND_ROBIN
      load_assignment:
        cluster_name: stepfun
        endpoints:
          - lb_endpoints:
              - endpoint:
                  address:
                    socket_address:
                      address: api.stepfun.com
                      port_value: 443
      transport_socket:
        name: envoy.transport_sockets.tls
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext
          "sni": "api.stepfun.com"

Request:

curl "http://localhost:10000/v1/chat/completions"  -H "Content-Type: application/json" -d '{
"model": "step-1-8k",
"messages": [
    {
        "role": "system",
        "content": "你是由阶跃星辰提供的AI聊天助手,你擅长中文,英文,以及多种其他语言的对话。在保证用户数据安全的前提下,你能对用户的问题和请求,作出快速和精准 的回答。同时,你的回答和建议应该拒绝黄赌毒,暴力恐怖主义的内容。"
    },
    {
        "role": "user",
        "content": "你好,请介绍一下阶跃星辰的人工智能!"
    }
]
}'

Response

{
  "id": "c0ab86c4213e81665861b1ef792377ef.018fc2def96479d5ab2211389dc00bbc",
  "object": "chat.completion",
  "created": 1716961343,
  "model": "step-1-8k",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "阶跃星辰是一家致力于提供人工智能技术的公司。他们的人工智能技术旨在为用户提供高效、精准的回答和建议。这种技术可以理解和处理多种语言,包括中文、英文以及其他一些语言。\n\n阶跃星辰的AI聊天助手能够快速理解用户的问题和请求,并提供相关的回答。它还能够保护用户的数据安全,确保用户的个人信息得到妥善处理。此外,这个AI聊天助手还能够拒绝黄赌毒、暴力恐怖主义等不良内容,为用户提供一个健康、积极的交流环境。\n\n总之,阶跃星辰的人工智能技术旨在为用户提供更好的交流体验,帮助他们解决问题、获取信息,并在这个过程中保护他们的数据安全。"
      },
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 85,
    "completion_tokens": 141,
    "total_tokens": 226
  }
}

Ⅴ. Special notes for reviews

Copy link
Collaborator

@CH3CHO CH3CHO left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Thanks.

@johnlanni
Copy link
Collaborator

@Chi-Kai Please resolve the conflicts

@johnlanni johnlanni merged commit ab1bc0a into alibaba:main Jun 19, 2024
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

AI 代理 Wasm 插件对接阶跃星辰
3 participants