Compliance tests and performance criteria for benchmarks #1505
mrasquinha-g
started this conversation in
General
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
Copying this over from the post mortem document ---
The problem here is not of TEST01 - rather the offline scenario is needing 24756 queries. As I understand this is not a proper requirement and we should choose this number specific to each benchmark. For example,
ResNet50 - 1024 (performance sample count) is enough as all inputs take the same time.
RetinaNet, bert, 3d-unet, gpt-j - dataset size as the processing time varies with the input
Beta Was this translation helpful? Give feedback.
All reactions