benchmarks: update JVM benchmarks #445
Conversation
|
Generated using |
|
/gcbrun exp -n dk-test-jvm3234 -m vertex_ai_gemini-1-5 -b jvm-small |
Changes the JVM benchmarks to be split into three buckets: 1) jvm-all: all java projects in oss-fuzz 2) jvm-medium: a smaller set of ~20 projects, which is useful for testing semantic changes in prompts without needing to check all projects. 3) jvm-small: set of three projects that can be used to test changes in infrastructure to ensure no infra regressions. Signed-off-by: David Korczynski <david@adalogics.com>
|
Experiment looks good |
| @@ -0,0 +1,55 @@ | |||
| "functions": | |||
| - "exceptions": [] | |||
There was a problem hiding this comment.
Is exceptions a new field in FI?
I noticed this new field in the new C/C++ benchmarks, too.
It is harmless to OFG, but if it serves no actual purpose, removing it would be cleaner.
There was a problem hiding this comment.
Nice catch. It's used but we need to bump introspector for this to reflect in the benchmarks generated from introspector.oss-fuzz.com versus locally. Doing this before landing this PR by way of google/oss-fuzz#12170
There was a problem hiding this comment.
The benchmarks have been updated now! Will run a small experiment and if all goes well there then I will run a large experiment
|
/gcbrun exp -n dk-test-infra5205 -m vertex_ai_gemini-1-5 -b jvm-small -i |
Signed-off-by: David Korczynski <david@adalogics.com>
|
Small experiment is looking great https://llm-exp.oss-fuzz.com/Result-reports/ofg-pr/2024-07-11-445-dk-test-infra5205-jvm-small/index.html, let's do a full run |
|
/gcbrun exp -n dk-test-infra5209 -m vertex_ai_gemini-1-5 -b jvm-all -i |
Changes the JVM benchmarks to be split into three buckets: 1) jvm-all: all java projects in oss-fuzz
2) jvm-medium: a smaller set of ~20 projects, which is useful for
testing semantic changes in prompts without needing to check all
projects.
3) jvm-small: set of three projects that can be used to test changes in
infrastructure to ensure no infra regressions.