Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: support groq ai model #967

Merged
merged 1 commit into from
May 17, 2024
Merged

feat: support groq ai model #967

merged 1 commit into from
May 17, 2024

Conversation

cr7258
Copy link
Collaborator

@cr7258 cr7258 commented May 17, 2024

Ⅰ. Describe what this PR did

Ⅱ. Does this pull request fix one issue?

fixes #955

Ⅲ. Why don't you add test cases (unit test/integration test)?

Ⅳ. Describe how to verify it

apiVersion: extensions.higress.io/v1alpha1
kind: WasmPlugin
metadata:
  name: ai-proxy-groq
  namespace: higress-system
spec:
  matchRules:
  - config:
      provider:
        type: groq
        apiTokens: 
          - "gsk_xxxxx"
    ingress:
    - groq
  url: oci://cr7258/ai-proxy:groq
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    higress.io/backend-protocol: HTTPS
    higress.io/destination: groq.dns
    higress.io/proxy-ssl-name: api.groq.com
    higress.io/proxy-ssl-server-name: "on"
  labels:
    higress.io/resource-definer: higress
  name: groq
  namespace: higress-system
spec:
  ingressClassName: higress
  rules:
  - http:
      paths:
      - backend:
          resource:
            apiGroup: networking.higress.io
            kind: McpBridge
            name: default
        path: /
        pathType: Prefix
---
apiVersion: networking.higress.io/v1
kind: McpBridge
metadata:
  name: default
  namespace: higress-system
spec:
  registries:
  - domain: api.groq.com
    name: groq
    port: 443
    type: dns
curl "http://localhost:10000/v1/chat/completions"  -H "Content-Type: application/json" -d '{
  "model": "llama3-8b-8192",
  "messages": [
    {
      "role": "user",
      "content": "你好,你是谁?"
    }
  ]
}'

# response
{
  "id": "chatcmpl-26733989-6c52-4056-b7a9-5da791bd7102",
  "object": "chat.completion",
  "created": 1715917967,
  "model": "llama3-8b-8192",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "😊 Ni Hao! (That's \"hello\" in Chinese!)\n\nI am LLaMA, an AI assistant developed by Meta AI that can understand and respond to human input in a conversational manner. I'm not a human, but a computer program designed to simulate conversations and answer questions to the best of my ability. I'm happy to chat with you in Chinese or help with any questions or topics you'd like to discuss! 😊"
      },
      "logprobs": null,
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 16,
    "prompt_time": 0.005,
    "completion_tokens": 89,
    "completion_time": 0.104,
    "total_tokens": 105,
    "total_time": 0.109
  },
  "system_fingerprint": "fp_dadc9d6142",
  "x_groq": {
    "id": "req_01hy2awmcxfpwbq56qh6svm7qz"
  }
}

Ⅴ. Special notes for reviews

@CH3CHO
Copy link
Collaborator

CH3CHO commented May 17, 2024

上面的 token 是可用的 token 吗?处于安全考虑已删除。

Copy link
Collaborator

@CH3CHO CH3CHO left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks.

@cr7258
Copy link
Collaborator Author

cr7258 commented May 17, 2024

上面的 token 是可用的 token 吗?处于安全考虑已删除。

@CH3CHO 谢谢,我前面忘记去掉了 😂

Copy link
Collaborator

@johnlanni johnlanni left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@johnlanni johnlanni merged commit 31242c3 into alibaba:main May 17, 2024
11 checks passed
@johnlanni
Copy link
Collaborator

@cr7258 可以帮忙更新下插件readme,增加groq支持的说明

@cr7258
Copy link
Collaborator Author

cr7258 commented May 17, 2024

@johnlanni 我直接把 ai-proxy 插件的说明更新到 higress 的官网上怎么样?另外我看其他新增的一些插件官网上也没有说明,如果需要的话,我后面也可以把其他插件的文档整理到官网上。

@johnlanni
Copy link
Collaborator

@cr7258 可以的 感谢🙏

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

AI 代理 Wasm 插件对接 Groq
3 participants