Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow placing user.conf under results/ #57

Open
psyhtest opened this issue Sep 9, 2020 · 0 comments
Open

Allow placing user.conf under results/ #57

psyhtest opened this issue Sep 9, 2020 · 0 comments

Comments

@psyhtest
Copy link

psyhtest commented Sep 9, 2020

At the moment, we place user.conf under measurements/ e.g. closed/Intel/measurements/clx_9282-2s_openvino-linux/ssd-small/Server. It can be argued, however, that it belongs under results e.g. closed/Intel/results/clx_9282-2s_openvino-linux/ssd-small/Server.

I don't think we should strictly require all performance and accuracy runs to happen with exactly the same LoadGen parameters. For example, the submitter may decide to do an accuracy Server run under a higher QPS than any of the corresponding performance runs to increase the throughput. (We don't measure the latency for accuracy runs, therefore latency constraints don't have to be obeyed.)

As another example, the submitter may have a bunch of VALID performance Server runs at slightly different QPS values e.g. [ 99.1, 99.2, 99.5, 99.3, 99.1 ]. In principle, they should have no problem to obtain 3 more VALID runs at QPS=99.1, but it would be just contributing to global warming. Instead, they should be allowed to submit these 5 runs and claim QPS=99.1 as achieved.

In such cases, the user.conf files may be slightly different. It's probably undecidable which one should be stored under measurements then. However, we won't be loosing any information if we allow to store them next to the LoadGen logs for each run.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant