[Meta-issue]: AutoGenBench Work Items

[afourney]

AutoGenBench Work Items

Small tasks (good first issues)

Beta Give feedback

1. Add a --dry-run option
2. Decide on proper casing of AutoGenBench folder names, and make changes as needed. #1424
   proj-autogenbench research +2
   

Immediate Priority

Beta Give feedback

1. Greatly enhance telemetry and logging to cover timestamp events, and track costs #1368
   +0
   

High Priority

Beta Give feedback

1. Allow users to quickly list and re-run scenarios that failed for non-evaluation-related technical reasons (e.g, Timeout errors)
2. Provide general purpose scripts for computing metrics such as costs, conversation statistics (# messages, etc.), detectable adverse events like loops, etc.
3. Coordinate with AutoGen Studio on logging format and metrics scripts/definitions
4. Include more test cases from MATH with a more principled subsample criterion #1054
   0.2 evaluation +2
   
5. Testbed - Clone AutoGPT test cases during runtime #1055
   proj-autogenbench +1
   

Longer Term Priorities

Beta Give feedback

1. Provide the ability to centrally host the results of runs/experiments (e.g., a dashboard-like experience for metrics as well as logs/traces)
2. Provide a central location for running experiments.
3. Set a cadence for releasing benchmarking products to the AutoGen team (e.g., weekly, prior to dot-releases, prior to major releases, etc)
4. Explore options for running benchmark tasks in parallel

Benchmarks to Possibly Onboard

Beta Give feedback

1. MMLU
2. GSM8k
3. WebArena
4. MiniWoB++

Wildcard Priorities (Unclear how we should support)

Beta Give feedback

1. Should the Testbed also cover running other non-AutoGen agents?

Completed

Click to expand

Completed tasks

Beta Give feedback

1. Update cli to print autogenbench.__version__ in help messages and in response to --version #1458
   +0
2. Update timestamp.txt generation to include autogenbench version #1461
   +0
3. Rename run_scenarios to autogenbench, and create python cli package #995
   +0
4. Standardize folder format for benchmarks (Move benchmark-specific READMEs, util scripts, metrics, etc. to appropriate folders) #976
   proj-autogenbench +1
   

   
5. Write blogpost outlining how to use the Testbed, what benchmarks are currently supported, our roadmap, and how to onboard new ones
6. Remove any 3rd-party problem sets from the repo. These should be retrieved and converted via download scripts.
7. Allow users to specify the Docker image when running scenarios. This makes setup faster, and many dependencies can be pre-loaded #985
   proj-autogenbench +1
   
8. Add the ability to run a subsample of a benchmark via the autogenbench run command. #1123
   +0
   
9. Testbed - deprecate use of the ENV file, and instead use the environment parameter of container.run() #987
   +0
   
10. Enable run_scenarios.py to read tasks from stdin, and decouple from strict folder layout so that it can be run as a command-line tool from anywhere. #996
    +0

[ph-ausseil]

@afourney :

Wildcard Priorities (Unclear how we should support)

Should the Testbed also cover running other Non-AutoGen agents?

Definitely, there is probably much more to learn that way but it ay also be the only part of the project that survives years. Agents will be replaced by better agent and so on... Ah on the "how", release an API & specs or adopt market ones

[afourney]

Will post an updated roadmap in the future.