-
Notifications
You must be signed in to change notification settings - Fork 263
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Minor unitxt fixes #2553
Minor unitxt fixes #2553
Conversation
@@ -15,7 +15,7 @@ def get_unitxt_spec(**kwargs) -> RunSpec: | |||
name_suffix = ",".join([f"{key}={value}" for key, value in kwargs.items()]) | |||
name = f"unitxt:{name_suffix}" | |||
scenario_spec = ScenarioSpec(class_name="helm.benchmark.scenarios.unitxt_scenario.UnitxtScenario", args=kwargs) | |||
adapter_spec = get_generation_adapter_spec() | |||
adapter_spec = get_generation_adapter_spec(max_train_instances=0) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can we set this things here as well @yifanmai ?
method: str = ""
global_prefix: str = ""
instructions: str = ""
input_prefix: str = ""
input_suffix: str = ""
reference_prefix: str = "A. "
reference_suffix: str = ""
output_prefix: str = ""
output_suffix: str = ""
instance_prefix: str = ""
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If possible can you address my comment on the code? This will be really useful for us. otherwise just ignore this
@elronbandel thanks for the advice. I fixed the Also note that you might still see weird prefixes and suffixed if you use certain Google / OpenAI / Mistral models, this will be fixed by #2550. |
Thanks @yifanmai ! |
max_train_instances
to 0 because unitxt handles training instance selectionAdapterSpec
for unitxtrun_entries_unitxt.conf
.gitignore
to only ignore configuration files at root