Skip to content
This repository has been archived by the owner on Sep 18, 2024. It is now read-only.

Merge master to v1.5 #2282

Merged
merged 2 commits into from
Apr 8, 2020
Merged

Merge master to v1.5 #2282

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
c61700f
Add doc of TextNAS (#2260)
pkuyym Apr 3, 2020
d2c5777
Add supported data types for PBT tuner (#2271)
RayMeng8 Apr 7, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
80 changes: 80 additions & 0 deletions docs/en_US/NAS/TextNAS.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,80 @@
# TextNAS

## Introduction

This is the implementation of the TextNAS algorithm proposed in the paper [TextNAS: A Neural Architecture Search Space tailored for Text Representation](https://arxiv.org/pdf/1912.10729.pdf). TextNAS is a neural architecture search algorithm tailored for text representation, more specifically, TextNAS is based on a novel search space consists of operators widely adopted to solve various NLP tasks, and TextNAS also supports multi-path ensemble within a single network to balance the width and depth of the architecture.

The search space of TextNAS contains:

* 1-D convolutional operator with filter size 1, 3, 5, 7
* recurrent operator (bi-directional GRU)
* self-attention operator
* pooling operator (max/average)

Following the ENAS algorithm, TextNAS also utilizes parameter sharing to accelerate the search speed and adopts a reinforcement-learning controller for the architecture sampling and generation. Please refer to the paper for more details of TextNAS.

## Preparation

Prepare the word vectors and SST dataset, and organize them in data directory as shown below:

```
textnas
├── data
│ ├── sst
│ │ └── trees
│ │ ├── dev.txt
│ │ ├── test.txt
│ │ └── train.txt
│ └── glove.840B.300d.txt
├── dataloader.py
├── model.py
├── ops.py
├── README.md
├── search.py
└── utils.py
```

The following link might be helpful for finding and downloading the corresponding dataset:

* [GloVe: Global Vectors for Word Representation](https://nlp.stanford.edu/projects/glove/)
* [glove.840B.300d.txt](http://nlp.stanford.edu/data/glove.840B.300d.zip)
* [Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank](https://nlp.stanford.edu/sentiment/)
* [trainDevTestTrees_PTB.zip](https://nlp.stanford.edu/sentiment/trainDevTestTrees_PTB.zip)

## Examples

### Search Space

[Example code](https://github.com/microsoft/nni/tree/master/examples/nas/textnas)

```bash
# In case NNI code is not cloned. If the code is cloned already, ignore this line and enter code folder.
git clone https://github.com/Microsoft/nni.git

# search the best architecture
cd examples/nas/textnas

# view more options for search
python3 search.py -h
```

After each search epoch, 10 sampled architectures will be tested directly. Their performances are expected to be 40% - 42% after 10 epochs.

By default, 20 sampled architectures will be exported into `checkpoints` directory for next step.

### retrain

```bash
# In case NNI code is not cloned. If the code is cloned already, ignore this line and enter code folder.
git clone https://github.com/Microsoft/nni.git

# search the best architecture
cd examples/nas/textnas

# default to retrain on sst-2
sh run_retrain.sh
```

## Reference

TextNAS directly uses EnasTrainer, please refer to [ENAS](./ENAS.md) for the trainer APIs.
1 change: 1 addition & 0 deletions docs/en_US/nas.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,5 +26,6 @@ For details, please refer to the following tutorials:
SPOS <NAS/SPOS>
CDARTS <NAS/CDARTS>
ProxylessNAS <NAS/Proxylessnas>
TextNAS <NAS/TextNAS>
Customize a NAS Algorithm <NAS/Advanced>
API Reference <NAS/NasReference>
6 changes: 5 additions & 1 deletion examples/nas/textnas/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,4 +42,8 @@ By default, 20 sampled architectures will be exported into `checkpoints` directo

## Retrain

Not ready.
```
sh run_retrain.sh
```

By default, the script will retrain the architecture provided by the author on the SST-2 dataset.
212 changes: 212 additions & 0 deletions examples/nas/textnas/arc/final_arc.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,212 @@
{
"LayerChoice1": [
false, false, false, false, false, true, false, false
],
"InputChoice2": [
true
],
"LayerChoice3": [
false, false, false, false, false, false, false, true
],
"InputChoice4": [
false
],
"InputChoice5": [
true, false
],
"LayerChoice6": [
false, false, false, true, false, false, false, false
],
"InputChoice7": [
false, false
],
"InputChoice8": [
false, false, true
],
"LayerChoice9": [
false, false, false, false, false, false, true, false
],
"InputChoice10": [
false, true, true
],
"InputChoice11": [
false, false, true, false
],
"LayerChoice12": [
false, true, false, false, false, false, false, false
],
"InputChoice13": [
false, true, false, false
],
"InputChoice14": [
false, false, false, false, true
],
"LayerChoice15": [
false, true, false, false, false, false, false, false
],
"InputChoice16": [
false, false, true, false, true
],
"InputChoice17": [
false, false, false, false, true
],
"LayerChoice18": [
true, false, false, false, false, false, false, false
],
"InputChoice19": [
false, false, true, true, true, true
],
"InputChoice20": [
true, false, false, false, false
],
"LayerChoice21": [
false, false, false, false, false, false, true, false
],
"InputChoice22": [
false, true, true, false, false, false, false
],
"InputChoice23": [
false, true, false, false, false
],
"LayerChoice24": [
false, false, false, false, false, true, false, false
],
"InputChoice25": [
false, true, false, true, true, false, true, true
],
"InputChoice26": [
false, false, true, false, false
],
"LayerChoice27": [
false, false, false, false, false, true, false, false
],
"InputChoice28": [
false, false, false, false, false, true, false, true, true
],
"InputChoice29": [
true, false, false, false, false
],
"LayerChoice30": [
false, false, false, false, false, false, false, true
],
"InputChoice31": [
true, true, false, false, true, false, false, true, true, false
],
"InputChoice32": [
true, false, false, false, false
],
"LayerChoice33": [
false, false, false, false, true, false, false, false
],
"InputChoice34": [
true, false, false, true, true, true, true, false, false, false, false
],
"InputChoice35": [
false, false, false, true, false
],
"LayerChoice36": [
false, true, false, false, false, false, false, false
],
"InputChoice37": [
true, true, false, true, false, true, false, false, true, false, false, false
],
"InputChoice38": [
false, false, false, true, false
],
"LayerChoice39": [
false, false, true, false, false, false, false, false
],
"InputChoice40": [
true, true, false, false, false, false, true, false, false, true, true, false, true
],
"InputChoice41": [
false, false, false, true, false
],
"LayerChoice42": [
true, false, false, false, false, false, false, false
],
"InputChoice43": [
false, false, true, false, false, false, true, true, true, false, true, true, false, false
],
"InputChoice44": [
false, false, false, false, true
],
"LayerChoice45": [
false, false, false, true, false, false, false, false
],
"InputChoice46": [
true, false, false, false, false, false, true, false, false, false, true, true, false, false, true
],
"InputChoice47": [
false, false, false, true, false
],
"LayerChoice48": [
false, false, true, false, false, false, false, false
],
"InputChoice49": [
false, false, false, false, false, false, false, false, false, true, true, false, true, false, true, false
],
"InputChoice50": [
false, false, false, false, true
],
"LayerChoice51": [
false, false, false, false, true, false, false, false
],
"InputChoice52": [
false, true, true, true, true, false, false, true, false, true, false, false, false, false, true, false, false
],
"InputChoice53": [
false, false, true, false, false
],
"LayerChoice54": [
false, false, false, true, false, false, false, false
],
"InputChoice55": [
false, false, false, false, false, true, false, false, false, false, false, false, false, true, true, true, false, true
],
"InputChoice56": [
false, false, true, false, false
],
"LayerChoice57": [
false, false, false, true, false, false, false, false
],
"InputChoice58": [
false, false, false, true, false, false, false, false, false, false, true, false, false, false, true, false, false, false, false
],
"InputChoice59": [
false, true, false, false, false
],
"LayerChoice60": [
false, false, false, false, false, true, false, false
],
"InputChoice61": [
true, true, false, false, false, false, false, false, false, false, true, true, false, false, true, true, true, true, false, false
],
"InputChoice62": [
true, false, false, false, false
],
"LayerChoice63": [
false, false, false, false, false, false, false, true
],
"InputChoice64": [
false, true, true, true, false, false, false, true, false, true, true, true, true, false, true, false, false, false, false, false, false
],
"InputChoice65": [
false, false, false, false, true
],
"LayerChoice66": [
false, false, false, false, false, false, false, true
],
"InputChoice67": [
false, false, true, true, true, true, false, true, false, true, true, false, false, false, false, true, false, false, false, false, false, true
],
"InputChoice68": [
false, false, false, true, false
],
"LayerChoice69": [
false, false, false, true, false, false, false, false
],
"InputChoice70": [
true, false, false, true, false, false, false, true, false, false, false, false, true, false, false, false, true, false, false, false, false, false, false
]
}
4 changes: 2 additions & 2 deletions examples/nas/textnas/run_retrain.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
export PYTHONPATH="$(pwd)"
export CUDA_VISIBLE_DEVICES=0

python -u retrain.py \
python3 -u retrain.py \
--train_ratio=1.0 \
--valid_ratio=1.0 \
--min_count=1 \
Expand Down Expand Up @@ -36,6 +36,6 @@ python -u retrain.py \
--child_lr_T_0=10 \
--child_lr_T_mul=2 \
--multi_path=True \
--child_fixed_arc="./checkpoints/architecture_00.json" \
--child_fixed_arc="./arc/final_arc.json" \
--fixed_seed=True \
"$@"
Empty file removed examples/trials/mnist-pbt-tuner-pytorch/__init__.py
View file
Open in desktop
Empty file.
4 changes: 2 additions & 2 deletions examples/trials/mnist-pbt-tuner-pytorch/mnist.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -155,8 +155,8 @@ def get_params():
help='learning rate (default: 0.01)')
parser.add_argument('--momentum', type=float, default=0.5, metavar='M',
help='SGD momentum (default: 0.5)')
parser.add_argument('--epochs', type=int, default=10, metavar='N',
help='number of epochs to train (default: 10)')
parser.add_argument('--epochs', type=int, default=1, metavar='N',
help='number of epochs to train (default: 1)')
parser.add_argument('--seed', type=int, default=1, metavar='S',
help='random seed (default: 1)')
parser.add_argument('--no_cuda', action='store_true', default=False,
Expand Down
Loading