Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open/Preview submissions from Inference v2.1 #140

Open
psyhtest opened this issue Dec 20, 2022 · 3 comments
Open

Open/Preview submissions from Inference v2.1 #140

psyhtest opened this issue Dec 20, 2022 · 3 comments

Comments

@psyhtest
Copy link

In the MLPerf Inference v2.1 round, Qualcomm and their partners submitted a number of RetinaNet results, which the Review Committee eventually accepted to the Open division under the Preview category. Also, ResNet50 and BERT results from the same systems were accepted to the Closed division under the Available category.

Given the extenuating circumstances surrounding these RetinaNet submissions, I believe it would be reasonable NOT to require ALL of such submissions to be replayed in the v3.0 round under the Available category, for example, if a system is no longer available for experimentation.

@rnaidu02
Copy link
Contributor

rnaidu02 commented Jan 3, 2023

Discussed two options:

  1. Submit in v3.0 and withdraw during review period so that preview results stay in v2.1 table.
  2. Not submitted in v3.0, and mark v2.1 preview results as retracted/invalid (exact action TBD based on the discussion from chairs meeting)

Option 2 was chosen

@rnaidu02
Copy link
Contributor

David is going to propose rule changes for Preview->Available

@psyhtest
Copy link
Author

This issue has been open for 1.5 years. Let's close it?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants