Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Various updates to AI Editor stuff #933

Merged
merged 9 commits into from
Sep 9, 2024
Merged

Various updates to AI Editor stuff #933

merged 9 commits into from
Sep 9, 2024

Conversation

richard-to
Copy link
Collaborator

@richard-to richard-to commented Sep 8, 2024 •
edited
Loading

Probably the best way to review this is to look at the individual commits (is there an easy way to chain pull requests?)

Changes:

  1. Added some new goldens
  2. Also renamed golden directories a bit. Added a timestamp suffix (used for roughly determining which goldens have not been fine tuned
  3. Added .env file support
  4. Added a way to include un-fine-tuned goldens to the prompt (quick way to sort of test the new goldens without doing a fine tuning run)
  5. Some minor refactoring (not really for ideal code structure, but the minimal amount of changes to include the adjustments added here
  6. Also add support for using Gemini client instead of OpenAI client (not using a fine tuned version yet, but bundling the goldens into the prompt for now)

@richard-to
Add more goldens
fa7be92
@richard-to
Add new uv dependencies
0aa352e 
- Allow usage of .env files to make it easier to manage env vars
- Add google-generativeai for testing gemini models
@richard-to
Rename golden dirs with timestamps
aa5138c 
Add timestamps so we can split goldens that have not been fine tuned and use them as part of the prompt.

Format is YYYYMMDDHHMM
@richard-to
Update service to create folders with timestamp suffix
4fbaab5
@richard-to
Update llm_lib to use include untuned examples
aa1afa7 
- Allow untuned examples to be used as part of the chat prompt
- Updates format goldens to generate the untuned examples subset
- Minor adjustments to prompt
@richard-to
Add another golden
de2f168
@richard-to
Update format goldens to create gemini CSV dataset
c17dbbd
@richard-to
Fix broken generate_run.py script
ead4ff4 
Some of the refactoring led to a regression. Add a quick fix.
@richard-to
Add gemini client support
3ff0d35
wwwillchen
wwwillchen approved these changes Sep 9, 2024
View reviewed changes
Copy link
Collaborator

@wwwillchen wwwillchen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM - btw, I'm prototyping a more structured way to manage evals and goldens, hope to send it out soon.

@richard-to richard-to merged commit 4d9f803 into google:main Sep 9, 2024
5 checks passed
@richard-to
Copy link
Collaborator Author

LGTM - btw, I'm prototyping a more structured way to manage evals and goldens, hope to send it out soon.

Nice. I'm looking forward to see that change. Definitely a lot of room for improvement there.

@richard-to richard-to deleted the ai-ai-ai branch September 28, 2024 18:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@wwwillchen wwwillchen wwwillchen approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@richard-to @wwwillchen