You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
At the moment, we place user.conf under measurements/ e.g. closed/Intel/measurements/clx_9282-2s_openvino-linux/ssd-small/Server. It can be argued, however, that it belongs under results e.g. closed/Intel/results/clx_9282-2s_openvino-linux/ssd-small/Server.
I don't think we should strictly require all performance and accuracy runs to happen with exactly the same LoadGen parameters. For example, the submitter may decide to do an accuracy Server run under a higher QPS than any of the corresponding performance runs to increase the throughput. (We don't measure the latency for accuracy runs, therefore latency constraints don't have to be obeyed.)
As another example, the submitter may have a bunch of VALID performance Server runs at slightly different QPS values e.g. [ 99.1, 99.2, 99.5, 99.3, 99.1 ]. In principle, they should have no problem to obtain 3 more VALID runs at QPS=99.1, but it would be just contributing to global warming. Instead, they should be allowed to submit these 5 runs and claim QPS=99.1 as achieved.
In such cases, the user.conf files may be slightly different. It's probably undecidable which one should be stored under measurements then. However, we won't be loosing any information if we allow to store them next to the LoadGen logs for each run.
The text was updated successfully, but these errors were encountered:
At the moment, we place
user.conf
undermeasurements/
e.g.closed/Intel/measurements/clx_9282-2s_openvino-linux/ssd-small/Server
. It can be argued, however, that it belongs underresults
e.g.closed/Intel/results/clx_9282-2s_openvino-linux/ssd-small/Server
.I don't think we should strictly require all performance and accuracy runs to happen with exactly the same LoadGen parameters. For example, the submitter may decide to do an accuracy Server run under a higher QPS than any of the corresponding performance runs to increase the throughput. (We don't measure the latency for accuracy runs, therefore latency constraints don't have to be obeyed.)
As another example, the submitter may have a bunch of VALID performance Server runs at slightly different QPS values e.g. [ 99.1, 99.2, 99.5, 99.3, 99.1 ]. In principle, they should have no problem to obtain 3 more VALID runs at QPS=99.1, but it would be just contributing to global warming. Instead, they should be allowed to submit these 5 runs and claim QPS=99.1 as achieved.
In such cases, the
user.conf
files may be slightly different. It's probably undecidable which one should be stored undermeasurements
then. However, we won't be loosing any information if we allow to store them next to the LoadGen logs for each run.The text was updated successfully, but these errors were encountered: