Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add SambaLingo Thai models #2747

Merged
merged 1 commit into from
Jun 17, 2024
Merged

Add SambaLingo Thai models #2747

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 19 additions & 0 deletions src/helm/config/model_deployments.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1021,6 +1021,25 @@ model_deployments:
client_spec:
class_name: "helm.clients.huggingface_client.HuggingFaceClient"

# SambaNova
- name: huggingface/sambalingo-thai-base
model_name: sambanova/sambalingo-thai-base
tokenizer_name: sambanova/sambalingo-thai-base
max_sequence_length: 4096
client_spec:
class_name: "helm.clients.huggingface_client.HuggingFaceClient"
args:
pretrained_model_name_or_path: sambanovasystems/SambaLingo-Thai-Base

- name: huggingface/sambalingo-thai-chat
model_name: sambanova/sambalingo-thai-chat
tokenizer_name: sambanova/sambalingo-thai-base
max_sequence_length: 4096
client_spec:
class_name: "helm.clients.huggingface_client.HuggingFaceClient"
args:
pretrained_model_name_or_path: sambanovasystems/SambaLingo-Thai-Base

## SCB10X
- name: huggingface/typhoon-v1.5-72b
model_name: scb10x/typhoon-v1.5-72b
Expand Down
19 changes: 19 additions & 0 deletions src/helm/config/model_metadata.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2350,6 +2350,25 @@ models:
release_date: 2022-03-25
tags: [] # TODO: add tags

# SambaNova
- name: sambanova/sambalingo-thai-chat
display_name: SambaLingo-Thai-Base
description: SambaLingo-Thai-Base is a pretrained bi-lingual Thai and English model that adapts Llama 2 (7B) to Thai by training on 38 billion tokens from the Thai split of the Cultura-X dataset. ([paper](https://arxiv.org/abs/2404.05829))
creator_organization_name: SambaLingo
access: open
num_parameters: 7000000000
release_date: 2024-04-08
tags: [TEXT_MODEL_TAG, PARTIAL_FUNCTIONALITY_TEXT_MODEL_TAG]

- name: sambanova/sambalingo-thai-chat
display_name: SambaLingo-Thai-Chat
description: SambaLingo-Thai-Chat is a chat model trained using direct preference optimization on SambaLingo-Thai-Base. SambaLingo-Thai-Base adapts Llama 2 (7B) to Thai by training on 38 billion tokens from the Thai split of the Cultura-X dataset. ([paper](https://arxiv.org/abs/2404.05829))
creator_organization_name: SambaLingo
access: open
num_parameters: 7000000000
release_date: 2024-04-08
tags: [TEXT_MODEL_TAG, PARTIAL_FUNCTIONALITY_TEXT_MODEL_TAG, INSTRUCTION_FOLLOWING_MODEL_TAG]

# SCB10X
- name: scb10x/typhoon-v1.5-72b
display_name: Typhoon v1.5 (72B)
Expand Down
9 changes: 9 additions & 0 deletions src/helm/config/tokenizer_configs.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -420,6 +420,15 @@ tokenizer_configs:
end_of_text_token: "<|endoftext|>"
prefix_token: ""

# SambaLingo
- name: sambanova/sambalingo-thai-base
tokenizer_spec:
class_name: "helm.clients.huggingface_client.HuggingFaceClient"
args:
pretrained_model_name_or_path: sambanovasystems/SambaLingo-Thai-Base
end_of_text_token: "</s>"
prefix_token: "<s>"

# Snowflake
- name: snowflake/snowflake-arctic-instruct
tokenizer_spec:
Expand Down