WebUI, NasUI Enhancement Backlog

[ultmaster]

July Candidate Items

• Adopt a whole new hyper-parameter chart. @ultmaster
• Improving code and data model structure. @ultmaster
• Adopt a whole new intermediate result chart. @ultmaster
• Refactor trial jobs table. @ultmaster
• Pagination. @colorjam
• Searching by parameters is too difficult to use. (v1.3 feedback) @colorjam
• Log files drawer cannot be closed via clicking overlay area. @Lijiaoa
• Weird spacing, e.g., top margin of "display top 10 trials". @Lijiaoa
• Support metrics other than default in intermediate result chart. Support dict metrics (switching keys) in intermediate result charts #2661 @Lijiaoa
• Weird capitalization, e.g., "Top Maximal trials". @Lijiaoa
• Unable to deal with infinity. (discussed in meeting 07/13) @Lijiaoa
• Show waiting trials in overview page. (v1.0 feedback) @yangou1988
• Update duration and max trial numbers. @yangou1988
• Should we make all columns center aligned? @yangou1988

Backlog

WebUI

Overview

• Log files drawer cannot be closed via clicking overlay area. @Lijiaoa
• Weird spacing, e.g., top margin of "display top 10 trials". @Lijiaoa
• Weird capitalization, e.g., "Top Maximal trials". @Lijiaoa
• Show waiting trials in overview page. (v1.0 feedback) @yangou1988
• Update duration and max trial numbers. @yangou1988

Default metric

• Zooming in is weird.

Hyper-parameters Parallel Coordinates

@ultmaster

• Unable to deal with unsupported search space and unmatched search space (with trials). [need to discuss] If experiment's search space doesn't match trial parameters #2668
• Unable to drag axes. Add log for uploading file in PAI mode #2642
• Support nested search space. WebUI nested search space enhancement #2555
• Customizable search space axes and metrics axes. WebUI nested search space enhancement #2555 Hyperparameter graph: val_loss as metric #1483
• The second choice type can not select data from yAxis when search space have 2 choice types. The second choice type can not select data from yAxis when search space have 2 choice type #1398

Trial duration

• Is this chart really useful? Or do users rather want to see the timeline?

Intermediate Result Chart

• Unable to deal with infinity. (discussed in meeting 07/13) @Lijiaoa
• Compare trials: incorrect data. Show 4 lines when selecting 3 trials. Unreproducible. [Exploring Test] Intermediate result graph for each trial got wrong when trial has many intermediate results #2568
• Support metrics other than default. Support dict metrics (switching keys) in intermediate result charts #2661 @Lijiaoa
• Lines are weirdly hidden when the chart is zoomed in. @ultmaster
• Customize the scale that is used for a specific metric. Logarithmic axes #1566 @ultmaster

Trial Jobs Table

@ultmaster

• Trial ID cannot be copied. should support to copy trial id in the table #2672
• Sort by additionally added keys. Support additional column sort #2580
• Collapse all and expand all. Could control the expansion of all trials' open row #1465
• Separate click event from selection event.
• Pagination. @colorjam
• Should we make all columns center aligned? @yangou1988
• Show finished trial logs. (needs API support from backend)
• Searching by parameters is too difficult to use. (v1.3 feedback) @colorjam

Connection between charts

• When certain trials get filtered out, the table should only show those trials.
• Redirect to corresponding trials in table when certain trials have been selected.

Misc

• Auto refresh doesn't save state.

Code and Development Experience

• No mock API server. Developers have to wait for the experiment to run into a certain state.
• Redundant logic for search space parsing.
• Naming is too complex and confusing. e.g., acc vs accuracy, parameters vs description.parameters, TableObj vs TableRecord.
• Redundant logic for legend and deprecated features, e.g., multi-phase. The support was actually quite limited back then and almost useless now.
• Upgrade to Fluent UI (originally Fabric UI).

NasUI

Known Issues

• Performance issue.
• Cannot focus on the key components, i.e., searched components, in the whole network.
• Support for structures are limited, e.g., recognition of the parameters in Conv2d.
• Scalability to different applications.

Code and Development Experience

• Introducing extra effort to maintain a separate codebase from WebUI.

[Lijiaoa]

@ultmaster Could you help update this issue status? Looks like many items had done...

[kvartet]

@ultmaster Could you help update this issue status? Looks like many items had done... (again