Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Evaluate Anthropic prompt formatting #437

Closed
teetone opened this issue May 16, 2022 · 5 comments
Closed

Evaluate Anthropic prompt formatting #437

teetone opened this issue May 16, 2022 · 5 comments
Labels
General p3 Priority 3 (do someday)

Comments

@teetone
Copy link
Member

teetone commented May 16, 2022 •
edited
Loading

Anthropic recommended we change the format:

Question: ...
Answer: ...

to


Human: What is the answer to "..."?

Assistant: The answer is "...".

(double newline preceding each speaker label)

We should try this new format when querying the Anthropic model on a few datasets and see if it significantly increases performance. If it doesn't improve performance, can we just forgo this (for simplicity and uniformity) and mention it in the appendix of our paper (cc @percyliang)?
@teetone teetone added p1 Priority 1 (Required for release) do before next big run labels May 16, 2022
@percyliang percyliang added p2 Priority 2 (Good to have for release) and removed p1 Priority 1 (Required for release) labels May 31, 2022
@rishibommasani
Copy link
Collaborator

Moving to P3 since this is nice-to-have.

@rishibommasani rishibommasani added p1 Priority 1 (Required for release) p3 Priority 3 (do someday) and removed p2 Priority 2 (Good to have for release) do before next big run p1 Priority 1 (Required for release) labels Jun 8, 2022
@rishibommasani rishibommasani mentioned this issue Aug 11, 2022
Closed
@percyliang percyliang added p2 Priority 2 (Good to have for release) and removed p3 Priority 3 (do someday) labels Sep 20, 2022
@percyliang
Copy link
Contributor

Add to run_specs_ablations.conf using the prompt=human_assistant RunExpander.

@rishibommasani
Copy link
Collaborator

@percyliang Going to assign for you; I will try to take since you have a bunch of stuff, but let me first handle some other things. Anyways imo this is not actually a P2 but a P3 that we just happen to be well-positioned to resolve.

@rishibommasani rishibommasani mentioned this issue Oct 3, 2022
Closed
@percyliang percyliang removed their assignment Oct 23, 2022
@percyliang percyliang added p3 Priority 3 (do someday) and removed p2 Priority 2 (Good to have for release) labels Oct 23, 2022
@percyliang
Copy link
Contributor

Moving to p3 since Anthropic-LM seems to work well anyway.

@yifanmai
Copy link
Collaborator

Closing.

@JosselinSomervilleRoberts already added this for Anthropic Claude in #1484 and #1862 because it is mandatory (Claude returns an error otherwise).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
General p3 Priority 3 (do someday)
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@yifanmai @percyliang @teetone @rishibommasani