plugin framework

[oliver-sanders]

Note: I've tagged this against 8.0.0 as it would be better if we could get any interface changes to the plugin
system out of the way before the first official release, however, this is not a critical feature and could be delayed
to 8.x.

If we do delay this make sure to add a warning to the plugin docs making it clear that the interface is
likely to change in the near future.

We have made our first tentative steps with plugins. I think this is a positive change both to the internal structure of Cylc and to its future with lots of exciting possibilities to tap down the line.

For now we have started implementing plugins using Python entry points. These do the job fairly well, they are easy to write and easy to discover, however, they are not easy to configure.

All plugins will be run if they are installed, the main loop plugins get around this by adding their own configuration [scheduler][main loop]plugins which can be added to on the CLI. Each main loop plugin also gets assigned a section in the config. This does the job but we really wouldn't want to go through this for each and every plugin.

Some things the current approach is lacking:

• Configuring plugins
  • Would be nice if each plugin got a default config namespace they could work with.
  • e.g. [scheduler][plugins][<plugin name>].
• Handling conflicting plugins
  • e.g. replace the built-in queue plugin with my custom one.
• Deactivating plugins (related to the previous)
  • e.g. turn off plugin X for this workflow.
• Allowing call order to be defined in the plugin framework
  • e.g. pluggy has a call-first/last option.
• Defining multi-faceted plugins.
  • e.g. a plugin might need to work with multiple entry points
• Iterating and executing plugins
  • Plugins should catch and report error in some standard way.
  • All the enable/disable logic should be gone in a standard way.
  • Would be nice to centralise this functionality.

We should consider adopting a plugin framework like pluggy.

A good example of pluggy in action is pytest, however, Cylc is different from this (any many other good plugin examples) in that cylc is not a single command, it is many and not all plugins are relevant to all functionalities. We don't want to waste time in each cylc invocation loading plugins that we don't need (e.g. you don't need to load the main loop plugins in order to run cylc validate). Cylc, being distributed has the problem that plugins might not even be installed in all places.

Something along the lines of this would be nice, however, would load more than we would like for each command invocation (must import the plugin and the things the plugin imports):

Something like this would be nice, however, the whole plugin would have to be loaded for each cylc invocation. (That might just be the price of opening up the config to modification, especially if we allow specifying defaults in the global configuration. Note that at present all plugins are imported on use thanks to the entry point mechanism.)

import cylc.plugin
from cylc.config import ConfigNode as Conf

@cylc.plugin.queue(replaces_built_in=True)
class MyQ:
    # the bit that should only get loaded when we configure queues
    ...

@cylc.plugin.main_loop.periodic
async def report_on_queue():
    # the bit that should only get loaded when we run workflows
    # NOTE: some plugins have async interfaces
    ...

@cylc.plugin.plugin
def setup(app):
    # the bit that always gets loaded by Cylc
    with app.configuration:
        Conf('max queue length', ...)
        Conf('priority order', ...)

Questions:

1. Is there a plugin framework that would be useful to us or would it just be easier to DIY?
2. Are we happy with each plugin getting an implicit section under [scheduler][plugin][<plugin name>]?
3. Are we happy to extend that to the global config too?
4. What about non [scheduler] plugins, do we need an [install][plugins] section too?
5. Can we realistically delay the import of plugins whilst enabling them to configure things like the config?

[hjoliver]

1. Is there a plugin framework that would be useful to us or would it just be easier to DIY?

TBD (should use one if there is one, obviously)

2. Are we happy with each plugin getting an implicit section under [scheduler][plugin][]?

That's probably the best strategy IMO. We definitely need plugins to be able to alter the config, and having plugin-specific config sections would be clean and clear. Is there a case for allowing them to alter existing config too, if necessary? (If so, the existing config should be a built-in plugin?).

3. Are we happy to extend that to the global config too?

Don't see why not.

4. What about non [scheduler] plugins, do we need an [install][plugins] section too?

Yes, probably.

5. Can we realistically delay the import of plugins whilst enabling them to configure things like the config?

TBD

[oliver-sanders]

BTW my thoughts:

2. Are we happy with each plugin getting an implicit section under [scheduler][plugin][]?

Yes, at least all "scheduler" plugins get a [scheduler][plugin] section (e.g. queue modules, main-loop plugins. xtriggers?, etc).

All "install" plugins get a [install][plugin] section, so on so forth.

3. Are we happy to extend that to the global config too?

Yes

4. What about non [scheduler] plugins, do we need an [install][plugins] section too?

Yes see (2).

[oliver-sanders]

Using a framework is a good idea, just need to find one to suit, changing from question into investigation.

Note we are currently using entry-points as a light-weight plugin interface. It has been suggested that it would be a good idea to pick an approach that doesn't mandate the plugins to be pip installed (as entry points do) as this prohibits bundling plugins with workflows which is an approach which makes sense for some cases. For example, look at the approach used by Sphinx-docs which uses a setup() function to register plugins. Note that Jinja2 globals/filters and Xtriggers can both be seen as Cylc plugins, whilst there may be advantages to pip installing them (i.e. allowing them to have dependencies) there are plenty of cases where you might want a quick single-use plugin bundled with a workflow rather than installed for all users.

However, see also #3780 which would potentially overcome this issue by allowing plugins to be pip installed in a per-workflow virtual environment.