NAS Benchmark #2578
NAS Benchmark #2578
Conversation
ultmaster
commented
Jun 18, 2020
•
ultmaster
commented
- Add NAS-Bench-201
- Add NAS-Bench-101
- NDS
- Test end-to-end installation script
- Test API
- Documentation
examples/nas/benchmark/README.md
Outdated
| ### Preparation | ||
|
|
||
| ```bash | ||
| docker run -e NNI_VERSION=${NNI_VERSION} -v ${HOME}/.nni/nasbenchmark:/outputs .:/root tensorflow/tensorflow:1.15.2-py3 /bin/bash /root/nasbench101.sh |
There was a problem hiding this comment.
is there docker image in this command?
There was a problem hiding this comment.
Docker image is tensorflow/tensorflow:1.15.2-py3
examples/nas/benchmark/README.md
Outdated
|
|
||
| Please replace `${NNI_VERSION}` with any NNI version, for example, v1.6 or master. | ||
|
|
||
| It takes about 70 minutes to dump the records and build index. Output size is about 1.9GB. |
There was a problem hiding this comment.
how can I store the generated dataset out of docker image?
There was a problem hiding this comment.
You mount a folder into /outputs in docker, and everything in that folder will be persisted.
examples/nas/benchmark/README.md
Outdated
|
|
||
| It takes about 70 minutes to dump the records and build index. Output size is about 1.9GB. | ||
|
|
||
| ### API Documentation |
There was a problem hiding this comment.
please complete this part
| return matrix | ||
|
|
||
|
|
||
| def nasbench_format_to_architecture_repr(adjacency_matrix, labeling): |
| @@ -0,0 +1,2 @@ | |||
| from .model import Nb201ComputedStats, Nb201IntermediateStats, Nb201RunConfig | |||
| from .shortcuts import query_nb201_computed_stats | |||
There was a problem hiding this comment.
what is the meaning of shortcuts?
There was a problem hiding this comment.
I followed the practice from some other projects. Do you have a better name in your mind?
docs/en_US/nas.rst
Outdated
| @@ -23,4 +23,5 @@ For details, please refer to the following tutorials: | |||
| One-shot NAS <NAS/one_shot_nas> | |||
| Customize a NAS Algorithm <NAS/Advanced> | |||
| NAS Visualization <NAS/Visualization> | |||
| NAS Benchmark <NAS/Benchmark> | |||
There was a problem hiding this comment.
Benchmark -> Benchmarks
| from .graph_util import hash_module, infer_num_vertices | ||
|
|
||
|
|
||
| def query_nb101_computed_stats(arch, num_epochs, isomorphism=True, reduction=None): |
There was a problem hiding this comment.
is "computed_stats" a common terminology?
| enumerate from 0. Since node 0 is input node, it is skipped in this dict. Each ``op`` | ||
| is one of :const:`nni.nas.benchmark.nasbench101.CONV3X3_BN_RELU`, | ||
| :const:`nni.nas.benchmark.nasbench101.CONV1X1_BN_RELU`, and :const:`nni.nas.benchmark.nasbench101.MAXPOOL3X3`. | ||
| Each ``input`` is a list of previous nodes. For example ``input5`` can be ``[0, 1, 3]``. |
There was a problem hiding this comment.
could you give a complete example of arch? it is fine to have only several nodes
