What does MutableScope mean?

[marsggbo]

what does mutables.MutableScope mean? when should we use this class? I see that it is only used in ENAS example, but enter_mutable_scope method is not overrided. Could you explain it, thx!

[QuanluZhang]

Thanks @marsggbo for asking. The detailed document for the design of NNI NAS interface and how to write a NAS algorithm on NNI is still ongoing, and will be complete in release v1.4

For your question, we introduce MutableScope in the interface because some NAS algorithms require structural/hierarchical information, for example, which LayerChoices and InputChoice are within a cell. So a MutableScope is like a tree, it tells which LayerChoice, InputChoice, MutableScope are under it.

For ENAS controller, it adds one more time step in every boundary of cells, whose hidden state represents the corresponding cell. Some NAS algorithms do not care about the structural information, then they can simply ignore them.

For your last question, @ultmaster will give you detailed answer.

[ultmaster]

In ENAS example, the mutable scope is used to construct the structured search space, and overriding callback functions is not needed.

enter_mutable_scope and exit_mutable_scope are called "callback functions", as they are triggered at the start and the end of __call__, which calls forward function in super().__call__:

nni/src/sdk/pynni/nni/nas/pytorch/mutables.py

Line 90 in bf2b929

return super().__call__(*args, **kwargs)

Up until today, these callback methods are designed but never used in code on master branch. If you ever want to use them, FYI, to implement a new mutator, there are two ways:

1. Inherit BaseMutator and implement a bunch of callbacks, like on_forward_layer_choice, enter_mutable_scope, each will be triggered at some place during the forward place. This is the way support proxylessnas with NNI NAS APIs #1863 adopted. As the control flow is implemented in a callback manner, this is for advanced users only. In this case, enter_mutable_scope and exit_mutable_scope is triggered on the enter and exit event of mutable scope. If your mutator can leverage that knowledge, you can implement them.
2. Inherit Mutator and implement only sample_search and sample_final, which corresponds to sampling an architecture for search, and reporting an architecture for export, respectively. In this case, you need to implement none of the callback functions. In this case, forward method is automatically based on your selected architecture. As the information of mutable scope is already embedded in the tree-structured search space, you can leverage that information when traversing this tree, thus overriding enter_mutable_scope and exit_mutable_scope is not needed.

Complementary to @QuanluZhang's reply, for this extra step, I think it's not mentioned in the paper, you can refer to enas code for details.