Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Refactor]: Refactor the evaluation directory #5222

Open
openhands-agent opened this issue Nov 23, 2024 · 3 comments · May be fixed by #5223
Open

[Refactor]: Refactor the evaluation directory #5222

openhands-agent opened this issue Nov 23, 2024 · 3 comments · May be fixed by #5223
Labels
fix-me Attempt to fix this issue with OpenHands

Comments

@openhands-agent
Copy link
Contributor

What problem or use case are you trying to solve?

Right now in the evaluation directory, the directory structure is very flat, and it is hard to tell which subdirectories are utilities related to implementing benchmarks or doing basic tests for openhands (utils, integration_tests, regression, static), and which are actual benchmarks from the ML literature (everything else).

To make this more clear, we can move all benchmarks to live under the evaluation/benchmarks/ directory. In addition, all other files that have to do with evaluation (including documentation, github workflows, etc.) will need to be checked and changed to maintain consistency.

While we do this, we can also add some of the benchmarks that are missing from the evaluation/README.md documentation.

@openhands-agent openhands-agent added enhancement New feature or request fix-me Attempt to fix this issue with OpenHands and removed fix-me Attempt to fix this issue with OpenHands labels Nov 23, 2024
@neubig
Copy link
Contributor

neubig commented Nov 23, 2024

Sorry, to clarify, this was me accidentally logged in as the openhands-agent account...

@neubig neubig added fix-me Attempt to fix this issue with OpenHands and removed enhancement New feature or request labels Nov 23, 2024
Copy link
Contributor

OpenHands started fixing the issue! You can monitor the progress here.

@All-Hands-AI All-Hands-AI deleted a comment from github-actions bot Nov 23, 2024
Copy link
Contributor

A potential fix has been generated and a draft PR #5223 has been created. Please review the changes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
fix-me Attempt to fix this issue with OpenHands
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants