Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Submitter and non-submitter definition for unverified result disclosures #157

Open
psyhtest opened this issue Jun 27, 2023 · 0 comments
Open

Comments

@psyhtest
Copy link

psyhtest commented Jun 27, 2023

The MLPerf Results Messaging Guidelines contain the following:

Submitters (i.e., those who have submitted results for review and verification by MLCommons) are not allowed to publish any results for a given benchmark version before its official publication date.

For example, if an organization only submits ResNet50 results to the upcoming v3.1 round, they are not allowed to disclose these results until the publication date. (However, they can seemingly obtain a similar set of results for a previous version of the benchmark e.g. v3.0, and disclose them as unverified results anytime they like, including during the v3.1 review period.)

Non-submitters (i.e., those who have not submitted results for review and verification by MLCommons) are not allowed to publish any results until two weeks after the official publication date for that benchmark version.

Now, if the same organization , who has only submitted ResNet50 results, obtains some unverified results, say, for GPT3, can they disclose them right after the official publication date, without waiting for two weeks?

On the one hand, the answer could be yes: this organization is a submitter, since they have submitted something to this round, just not this benchmark. On the other hand, the answer should be no: they have submitted nothing for this benchmark.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant