Skip to content
This repository has been archived by the owner on Sep 18, 2024. It is now read-only.

new feature : NAS-NLP-benchmark #3140

Merged
merged 16 commits into from
Dec 11, 2020
Merged

Conversation

98may
Copy link
Contributor

@98may 98may commented Nov 30, 2020

finish basic function, pass test with test_example.py.
Support query for both final and immediate training results.

@98may
Copy link
Contributor Author

98may commented Nov 30, 2020

Usage:
0. cd nni/examples/nas/benchmarks

  1. Install dependencies via pip3 install -r xxx.requirements.txt. xxx can be nlp.
  2. Generate the database via ./xxx.sh. The directory that stores the benchmark file can be configured with NASBENCHMARK_DIR environment variable, which defaults to ~/.nni/nasbenchmark.

@98may 98may marked this pull request as draft November 30, 2020 03:37
@98may 98may marked this pull request as ready for review November 30, 2020 04:35
@98may
Copy link
Contributor Author

98may commented Nov 30, 2020

TBD: check if query api should add an attribute called "reduction", which depends on if the paper's repo's "multi_run logs" is a superset of "single_run logs" .

@98may
Copy link
Contributor Author

98may commented Nov 30, 2020

the paper's repo's "multi_run logs(m)" is not a superset of "single_run logs(s)", it has some dupilicates as the following data declares:
len(m) = 12343
len(set(m)) = 12343
len(s) = 14323
len(set(s)) = 14323
len(set(m) & set(s)) = 4114.

Now it is ready for review.

nni/nas/benchmarks/nlp/db_gen.py Show resolved Hide resolved
nni/nas/benchmarks/nlp/test_example.py Outdated Show resolved Hide resolved
nni/nas/benchmarks/nlp/query.py Show resolved Hide resolved
nni/nas/benchmarks/nlp/model.py Show resolved Hide resolved
@98may 98may requested a review from ultmaster December 1, 2020 07:27
@ultmaster
Copy link
Contributor

Please fix the pipelines.

@QuanluZhang
Copy link
Contributor

@98may need to update doc accordingly


## NLP

[Paper link](https://arxiv.org/abs/2006.07116)     [Open-source](https://github.com/fmsnew/nas-bench-nlp-release)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

have you verified that the accuracy number can be reproduced with the authors' code?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not really. Do I need to double check that? If so, I will try to run the paper's repo's code.

@liuzhe-lz liuzhe-lz closed this Dec 11, 2020
@liuzhe-lz liuzhe-lz reopened this Dec 11, 2020
@liuzhe-lz liuzhe-lz merged commit 697c60e into microsoft:master Dec 11, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants