runner constructor

[densechen]

Add RunnerConstructor feature to build_runner.

It will not effect the original code feature, but just an enhancement.

Refer here for more details: issue1225

[CLAassistant]

All committers have signed the CLA.

[densechen]

import in __init__.py

[codecov]

Codecov Report

Merging #1238 (d3841d2) into master (1790e9f) will increase coverage by 0.03%.
The diff coverage is 90.00%.

❗ Current head d3841d2 differs from pull request most recent head dcabcff. Consider uploading reports for the commit dcabcff to get more accurate results

@@            Coverage Diff             @@
##           master    #1238      +/-   ##
==========================================
+ Coverage   68.23%   68.26%   +0.03%     
==========================================
  Files         160      161       +1     
  Lines       10721    10739      +18     
  Branches     1969     1971       +2     
==========================================
+ Hits         7315     7331      +16     
- Misses       3022     3023       +1     
- Partials      384      385       +1     

Flag	Coverage Δ
unittests	68.26% <90.00%> (+0.03%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
mmcv/runner/default_constructor.py	77.77% <77.77%> (ø)
mmcv/runner/__init__.py	100.00% <100.00%> (ø)
mmcv/runner/builder.py	100.00% <100.00%> (ø)

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 1790e9f...dcabcff. Read the comment docs.

[ZwwWayne]

Hi @densechen ，
Thanks for the PR. Could you show the new use cases used in your projects after introducing RunnerConstructor? The information would be beneficial for us to consider the new design.

[densechen]

Hi, @ZwwWayne , if a RunnerConstructor is provided, we can have the ability to make some extra operations on the build runner without the necessary to change the user code.
By the way, we have developed a new open source pytorch-style federated learning library, which will be released soon.
We are currently palning to write an extra package that constumed for mm-xxx project. It will make all the mm-xxx project enable to do federated learning experiments without the necessary to change the user code, but just import our extra package at the beginning of code.
While developing this extra package, we found that it may be more flexible to provide a RunnerConstructor in our case.

[zhouzaida]

Hi, @ZwwWayne , if a RunnerConstructor is provided, we can have the ability to make some extra operations on the build runner without the necessary to change the user code.
By the way, we have developed a new open source pytorch-style federated learning library, which will be released soon.
We are currently palning to write an extra package that constumed for mm-xxx project. It will make all the mm-xxx project enable to do federated learning experiments without the necessary to change the user code, but just import our extra package at the beginning of code.
While developing this extra package, we found that it may be more flexible to provide a RunnerConstructor in our case.

Got it. Thanks for your contribution. And could you provide an example to show how to use the RunnerConstructor

[densechen]

Currently, if we want to do something with runner provided by mmcv, we need such piece of code:

from mmcv.runner.builder import RUNNERS, build_runner
from openfed.common.gluer import glue
from typing import Any

from .builder import build_openfed

def openfed_runner(model,
                   batch_processor=None,
                   optimizer=None,
                   work_dir=None,
                   logger=None,
                   meta=None,
                   max_iters=None,
                   max_epochs=None,
                   runner_cfg=None):
    # Build Runner
    runner = build_runner(
        runner_cfg,
        default_args=dict(
            model=model,
            batch_processor=batch_processor,
            optimizer=optimizer,
            work_dir=work_dir,
            logger=logger,
            meta=meta,
            max_iters=max_iters,
            max_epochs=max_epochs,
        )
    )
    # DO SOMETHING HERE
    runner.openfed_name = 'XXX'
    return runner


@RUNNERS.register_module(name='OpenFedRunner')
class OpenFedRunner(object):
    def __init__(self, *args, **kwargs):
        super().__init__()
        self.openfed_runner = openfed_runner(*args, **kwargs)

    def __getattribute__(self, name: str) -> Any:
        if name == 'openfed_runner':
            return super().__getattribute__(name)
        return getattr(self.openfed_runner, name)

    def __setattr__(self, name: str, value: Any) -> None:
        if name == 'openfed_runner':
            return super().__setattr__(name, value)
        return setattr(self.openfed_runner, name, value)

This code is quite ugly.
But if a runner_constructor is provided, we only need to define a new runner_constructor for doing this.
It seems more beautiful and flexible.
We are looking forward it.

[zhouzaida]

got it

[zhouzaida]

hi, please fix the CI

[densechen]

fix yapf

[zhouzaida]

fix yapf

you can refer to https://github.com/open-mmlab/mmcv/blob/master/CONTRIBUTING.md

[densechen]

fix yapf

[densechen]

Waiting for merging
@zhouzaida

[zhouzaida]

Waiting for merging
@zhouzaida

hi @densechen , thanks for your contribution again. Before merging, An unittest should be added (refer to https://github.com/open-mmlab/mmcv/blob/master/tests/test_runner/test_optimizer.py). In addition, maybe we can enhance the docstring of DefaultRunnerConstructor like example or Args. BTW, DefaultRunnerConstructor is a really great design.

[ZwwWayne]

Overall, this PR looks good to me. We need to resolve conflict before the merge.

[densechen]

Hi, @zhouzaida Can we have a further discussion via WeChat? My Email is densechen@foxmail.com

[zhouzaida]

Hi, @zhouzaida Can we have a further discussion via WeChat? My Email is densechen@foxmail.com

Great, I send an email to you

[ZwwWayne]

This PR can be merged after resolving linting issues and conflicts.

[densechen]

Using Examples: Inject some new properties and functions for existing Runner, such as EpochBasedRunner, though a RunnerReconstructor.

1. Define a new RunnerReconstructor:

from mmcv.runner import RUNNER_BUILDERS

@RUNNER_BUILDERS.register_module()
class MyRunnerConstructor:
    def __init__(self, runner_cfg, default_args=None):
        if not isinstance(runner_cfg, dict):
            raise TypeError('runner_cfg should be a dict',
                            f'but got {type(runner_cfg)}')
        self.runner_cfg = runner_cfg
        self.default_args = default_args

    def __call__(self):
        runner = RUNNERS.build(self.runner_cfg, default_args=self.default_args)
        # Add new properties for existing runner
        runner.my_name = 'my_runner'
        runner.my_function = lambda self: print(self.my_name)
        ...

2. Use this RunnerConstructor, the config file is like:

runner = dict(type='EpochBasedRunner', max_epochs=40, constructor='MyRunnerConstructor')

[ZwwWayne]

1. need to resolve conflicts.
2. Would you like to also update the usage of the runner in the documentation? You can also do that in a later PR.

[zhouzaida]

LGTM

[densechen]

re-commit

[zhouzaida]

I am so sorry to force push to your master branch

[densechen]

recommit

[densechen]

#1296

[zhouzaida]

Thanks