Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Edge AI Benchmark: Consolidate/Prioritize metrics #118

Open
YuxiaoHu opened this issue Jul 7, 2021 · 2 comments
Open

Edge AI Benchmark: Consolidate/Prioritize metrics #118

YuxiaoHu opened this issue Jul 7, 2021 · 2 comments
Labels
kind/documentation Categorizes issue or PR as related to documentation.

Comments

@YuxiaoHu
Copy link

YuxiaoHu commented Jul 7, 2021

The draft benchmark version looks good, which is a nice start point. Thanks for the great efforts!

Issues: There are too many metrics, which lead to two problems:

  • Difficult for readers to understand each of them according to the short descriptions. This proposal should be self-contained, so that the readers won’t need to read all the references. It will be nice to add descriptions/examples/Figures for each of the metrics;
  • For new users, it will be difficult to execute. I suggest we start from one or two core metrics for each scenario (like P0 features), and provide additional metrics (as P1/P2, …) for advanced users or readers who are interested.
@YuxiaoHu YuxiaoHu added the kind/documentation Categorizes issue or PR as related to documentation. label Jul 7, 2021
@YuxiaoHu YuxiaoHu changed the title Edge AI Benchmark review: Consolidate/Prioritize metrics Edge AI Benchmark: Consolidate/Prioritize metrics Jul 7, 2021
@MooreZheng
Copy link

Thanks. We have updated the first draft of the specification.

The specification is still open to edit. Any interested member please also refer to #96

@MooreZheng
Copy link

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/documentation Categorizes issue or PR as related to documentation.
Projects
None yet
Development

No branches or pull requests

2 participants