NAS visualization #2085

ultmaster · 2020-02-20T15:52:02Z

~~This PR is open for code comparison against master. Not ready for review yet.~~

Working items:

Makefile
Fix UI fail bug when key is missing.

Might do if I have time:

Use proto graph and log in protobuf (instead of NodePy)
Configurable node elimination on UI
Configurable node collapse on UI
Display attributes of ops on UI

I will post some preview here when I think it's ready.

Ready for review

Deferred items:

These items might be done in separate PRs.

How to test:

Change DARTS search into something like this:

class MyTrainer(DartsTrainer):
    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self.status_writer = open("log", "w")

    def _logits_and_loss(self, X, y):
        self.mutator.reset()
        logits = self.model(X)
        loss = self.loss(logits, y)
        print(json.dumps(self.mutator.status()), file=self.status_writer)
        self.status_writer.flush()
        return logits, loss


model = CNN(32, 3, 16, 10, 8)
model.cuda()
mutator = DartsMutator(model)
vis_graph = mutator.graph(torch.randn((1, 3, 32, 32)).cuda())
with open("graph.json", "w") as f:
    json.dump(vis_graph, f)

dataset_train, dataset_valid = datasets.get_dataset("cifar10")
criterion = nn.CrossEntropyLoss()
optim = torch.optim.SGD(model.parameters(), 0.025, momentum=0.9, weight_decay=3.0E-4)
trainer = MyTrainer(model=model,
                    mutator=mutator,
                    loss=criterion,
                    metrics=lambda output, target: accuracy(output, target, topk=(1,)),
                    optimizer=optim,
                    num_epochs=1,
                    dataset_train=dataset_train,
                    dataset_valid=dataset_valid,
                    batch_size=64,
                    log_frequency=10,
                    arc_learning_rate=0.1,
                    unrolled=False)
trainer.train()

This will write the graph to graph.json before starts, and write each step as a line into a file called log.

Launch nnictl webui nas --logdir /path/to/the/directory/containing/two/files. Find your UI at port ~~6667~~ 6060.

Lijiaoa · 2020-02-25T01:41:49Z

src/nasui/src/Chart.tsx

+    if (graph === undefined || activation === undefined)
+      return;
+    const weights = graph.weightFromMutables(activation);
+    console.log(weights.size);


I thought could delete this line.

I will delete all console.log before merge.

Lijiaoa · 2020-02-25T01:43:27Z

src/nasui/src/Chart.tsx

+      if (graphChanged)
+        this.expandSet = lodash.cloneDeep(graph.defaultExpandSet);
+      const graphEl = this.graphElements();
+      console.log(graphEl);


could add no-console: true in your lint

Lijiaoa · 2020-02-25T01:47:54Z

src/nasui/package.json

+    "@bardit/cytoscape-expand-collapse": "^2.0.3",
+    "@material-ui/core": "^4.9.3",
+    "@material-ui/icons": "^4.9.1",
+    "@testing-library/jest-dom": "^4.2.4",
+    "@testing-library/react": "^9.3.2",
+    "@testing-library/user-event": "^7.1.2",
+    "@types/cytoscape": "^3.14.0",
+    "@types/d3-graphviz": "^2.6.3",
+    "@types/dagre-d3": "^0.4.39",
+    "@types/graphlib-dot": "^0.6.1",
+    "@types/jest": "^24.0.0",
+    "@types/lodash": "^4.14.149",
+    "@types/node": "^12.0.0",
+    "@types/react": "^16.9.0",
+    "@types/react-dom": "^16.9.0",


some of this dependencies maybe could put into devDependencies

It's in future item: Clean up unused dependencies.

Lijiaoa · 2020-02-25T01:50:34Z

src/nasui/src/App.tsx

+                <ListItem className={classes.listSubtitle}>Inputs ({info.inputs.length})</ListItem>
+                {
+                  info.inputs.map((item, i) => <ListItem className={classes.listItem} key={`input${i}`}>{item}</ListItem>)
+                }</React.Fragment>


</React.Fragment>换个行吧

Lijiaoa · 2020-02-25T01:52:32Z

src/nasui/src/App.tsx

+    const handleSliderChange = (event: ChangeEvent<{}>, value: number | number[]) => {
+      this.setState({ sliderValue: value as number });
+    };
+    const handleSettingsDialogToggle = (value: boolean) => () => {
+      this.setState({ settingsOpen: value });
+    };
+    const handleSettingsChange = (name: string) => (event: React.ChangeEvent<HTMLInputElement>) => {
+      this.setState({
+        ...this.state,
+        [name]: event.target.checked
+      }, () => {
+        this.setState({
+          graph: new Graph(this.state.graphData, this.state.hideSidechainNodes),
+        })
+      });
+    };


下个版本考虑把这些方法挪出render() 这个周期吗

Good suggestion. Will do.

Lijiaoa · 2020-02-25T01:55:45Z

src/nasui/src/Chart.tsx

+      'text-wrap': 'wrap',
+      'text-valign': 'center',
+      'text-halign': 'center',
+      'font-family': '"Roboto", "Helvetica", "Arial", sans-serif',


如果跟WebUI风格保持一致的话，也应该选Segoe UI作为第一字体？

I'm aligning with tensorboard. So I used Material-UI + Roboto.

Lijiaoa · 2020-02-25T01:56:28Z

src/nasui/src/Chart.tsx

+      'padding-left': '8px',
+      'padding-right': '8px',
+      'padding-top': '8px',
+      'padding-bottom': '8px',


padding: 8px;

Lijiaoa · 2020-02-25T01:57:22Z

src/nasui/src/Chart.tsx

+      'padding-top': '30px',
+      'padding-bottom': '10px',
+      'padding-right': '10px',
+      'padding-left': '10px',


也可以合成一条

I don't think so. Cytoscape use its own style format. I tried to merge them and it didn't work.

SparkSnail · 2020-02-27T02:25:35Z

src/nasui/src/App.tsx

+    const { classes } = this.props;
+    const { selectedNode, graph } = this.state;
+    if (graph === undefined)
+      return null;


what is the difference between null and undefined?

React suggests null when returning a None element. Fixed. Thanks.

SparkSnail · 2020-02-27T02:36:17Z

src/nasui/src/graphUtils.ts

+    }
+    this.defaultExpandSet = this.getDefaultExpandSet(graphData.mutable);
+    this.mutableEdges = this.inferMutableEdges(graphData.mutable);
+    console.log(this);


remove console.log()

SparkSnail · 2020-02-27T02:52:07Z

src/nasui/src/react-app-env.d.ts

@@ -0,0 +1 @@
+/// <reference types="react-scripts" />


Is this file useful?

@ultmaster better to explain why

It's explained in facebook/create-react-app#6560. I guess we should respect the practice of create-react-app.

SparkSnail · 2020-02-27T03:42:08Z

src/nasui/src/serviceWorker.ts

+      };
+    })
+    .catch(error => {
+      console.error('Error during service worker registration:', error);


use Alert(error)?

This file is auto-generated. I don't want to modify it.

SparkSnail · 2020-02-27T03:43:50Z

src/nasui/src/setupTests.ts

@@ -0,0 +1,5 @@
+// jest-dom adds custom jest matchers for asserting on DOM nodes.


Is this useful? It seems that there is no main logic code.

It's auto-generated. Removing it for now.

SparkSnail · 2020-02-27T03:47:34Z

tools/nni_cmd/nnictl.py

@@ -158,6 +158,10 @@ def parse_args():
    parser_webui_url = parser_webui_subparsers.add_parser('url', help='show the url of web ui')
    parser_webui_url.add_argument('id', nargs='?', help='the id of experiment')
    parser_webui_url.set_defaults(func=webui_url)
+    parser_webui_nas = parser_webui_subparsers.add_parser('nas', help='show nas ui')
+    parser_webui_nas.add_argument('--port', default=6060, type=int)


add help= to explain the argement.

QuanluZhang · 2020-02-28T04:13:05Z

src/sdk/pynni/nni/nas/pytorch/mutator.py

+        for mutable in self.mutables.traverse(deduplicate=False):
+            modules = mutable.name.split(".")
+            path = [
+                {"type": self.model.__class__.__name__, "name": ""}
+            ]
+            m = self.model
+            for module in modules:
+                m = getattr(m, module)
+                path.append({
+                    "type": m.__class__.__name__,
+                    "name": module
+                })
+            result["mutable"][mutable.key].append(path)


better to add comments to explain this part

QuanluZhang · 2020-02-28T04:16:15Z

tools/nni_cmd/nnictl.py

@@ -203,7 +207,7 @@ def parse_args():
    parser_tensorboard_start = parser_tensorboard_subparsers.add_parser('start', help='start tensorboard')
    parser_tensorboard_start.add_argument('id', nargs='?', help='the id of experiment')
    parser_tensorboard_start.add_argument('--trial_id', '-T', dest='trial_id', help='the id of trial')
-    parser_tensorboard_start.add_argument('--port', dest='port', default=6006, help='the port to start tensorboard')
+    parser_tensorboard_start.add_argument('--port', dest='port', default=6060, help='the port to start tensorboard')


why change the default value?

It's an accident. Sorry.

QuanluZhang · 2020-02-28T04:19:14Z

@ultmaster could you write doc for it, including how to change user code, how to start the nas ui. From the code, I can get how you get the super graph, but I did not get how the log files come from, how to generate those log files for nas ui.

ultmaster · 2020-02-28T04:25:47Z

@ultmaster could you write doc for it, including how to change user code, how to start the nas ui.

I introduced briefly how to use it in the description of this PR, but it's for test only.

The docs will be in next release. This is because currently, the only way to launch it is to build from source and launch it with nnictl under nni directly, which is because, release integration is not done.

BTW, as discussed in the last meeting, I need help to integrate it into release (including changing all the Makefiles and make it work on pypi version).

Ideally, graph logging should be registered as hook (callback) into trainer. But since on_batch_end is not implemented at all, we will only be able to dump status after each epoch.

ultmaster added 9 commits February 20, 2020 20:37

first version of sdk

297d781

update

245d51c

update

bb78db5

update

334cebd

update

267a0cf

update

d6da7c5

update

c48cb1b

update

1922cee

update

3851102

ultmaster marked this pull request as ready for review February 24, 2020 15:35

update

e94319f

ultmaster requested review from QuanluZhang, SparkSnail, scarlett2018, liuzhe-lz and lvybriage February 24, 2020 15:40

Lijiaoa reviewed Feb 25, 2020

View reviewed changes

SparkSnail reviewed Feb 27, 2020

View reviewed changes

ultmaster added 4 commits February 28, 2020 09:55

update

70838d4

remove unused dependencies

f4c5ce9

restore webui yarn.lock

bf2064a

update

bd39af0

QuanluZhang reviewed Feb 28, 2020

View reviewed changes

lvybriage approved these changes Feb 28, 2020

View reviewed changes

add comments

e6cfe4b

scarlett2018 mentioned this pull request Feb 28, 2020

Iteration Plan for release in Mar #2079

Closed

35 tasks

SparkSnail approved these changes Mar 2, 2020

View reviewed changes

QuanluZhang approved these changes Mar 3, 2020

View reviewed changes

ultmaster merged commit 9987014 into microsoft:master Mar 3, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

NAS visualization #2085

NAS visualization #2085

ultmaster commented Feb 20, 2020 •

edited

Loading

Lijiaoa Feb 25, 2020

ultmaster Feb 25, 2020

Lijiaoa Feb 25, 2020

Lijiaoa Feb 25, 2020

ultmaster Feb 25, 2020

Lijiaoa Feb 25, 2020

Lijiaoa Feb 25, 2020

ultmaster Feb 25, 2020

Lijiaoa Feb 25, 2020

ultmaster Feb 25, 2020

Lijiaoa Feb 25, 2020

ultmaster Feb 25, 2020

Lijiaoa Feb 25, 2020

ultmaster Feb 25, 2020

SparkSnail Feb 27, 2020

ultmaster Feb 28, 2020

SparkSnail Feb 27, 2020

ultmaster Feb 28, 2020

SparkSnail Feb 27, 2020

ultmaster Feb 28, 2020

QuanluZhang Feb 28, 2020

ultmaster Feb 28, 2020

SparkSnail Feb 27, 2020

ultmaster Feb 28, 2020

SparkSnail Feb 27, 2020

ultmaster Feb 28, 2020

SparkSnail Feb 27, 2020

ultmaster Feb 28, 2020

QuanluZhang Feb 28, 2020

ultmaster Feb 28, 2020

QuanluZhang Feb 28, 2020

ultmaster Feb 28, 2020

QuanluZhang commented Feb 28, 2020 •

edited

Loading

ultmaster commented Feb 28, 2020 •

edited

Loading

		@@ -0,0 +1,5 @@
		// jest-dom adds custom jest matchers for asserting on DOM nodes.

NAS visualization #2085

NAS visualization #2085

Conversation

ultmaster commented Feb 20, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

QuanluZhang commented Feb 28, 2020 • edited Loading

ultmaster commented Feb 28, 2020 • edited Loading

ultmaster commented Feb 20, 2020 •

edited

Loading

QuanluZhang commented Feb 28, 2020 •

edited

Loading

ultmaster commented Feb 28, 2020 •

edited

Loading