Package-ize core plugins

[dgw]

Todo list, created for 8.x dev cycle based on 7.x core plugins:

• admin core bot-management, should stay part of core
• adminchannel
  Pros and cons to both keeping this as core and extracting it.
• announce
• bugzilla: removed from core in 8.0 (bugzilla: remove module extracted to separate package #2481); moved to https://github.com/sopel-irc/sopel-bugzilla and published to PyPI as sopel-bugzilla
• calc
• choose
• clock — the non-API-related stuff only; see Adopt clock features used by core into core #2467
• countdown
• currency
• dice
• emoticons
• find
• find_updates probably stay part of core
• help: due to be removed in 8.0 (help: replace with dependency on sopel-help #2332); extended version available at https://github.com/sopel-irc/sopel-help and published to PyPI as sopel-help
• instagram: removed from core in 7.1 (instagram: remove fragile plugin #2000), moved to https://github.com/sopel-irc/sopel-instagram in non-functional state in case it can be revived later (no PyPI package published)
• invite: possibly stay as part of core
• ip: removed from core for 8.0 (ip: remove plugin extracted to external package #2523); moved to https://github.com/sopel-irc/sopel-iplookup and published to PyPI as sopel-iplookup
• ipython: removed from core in 7.0 (ipython: remove plugin (moving to separate package) #1684); moved to https://github.com/sopel-irc/sopel-ipython and published to PyPI as sopel-ipython
• isup
• lmgtfy
• meetbot: removed from core in 8.0 (meetbot: remove in favor of external package #2477); moved to https://github.com/sopel-irc/sopel-meet bot and published to PyPI as sopel-meetbot
• ping
• pronouns
  Could stay as part of core if core starts making use of pronouns anywhere. Currently it's just a way to set your own pronouns so others can query the plugin if they don't know how to address you.
• py: removed from core in 8.0 (py, build: drop py plugin from core & require external package #2411); moved to https://github.com/sopel-irc/sopel-py and published to PyPI as sopel-py
• rand
• reddit: removed from core in 8.0 (reddit: extract to external package #2444); moved to https://github.com/sopel-irc/sopel-reddit and published to PyPI as sopel-reddit
• reload: should stay part of core, but shed the .update command that doesn't belong
• remind: removed from core in 8.0 (remind: remove plugin reimplemented as external package w/better tests #2478); moved to https://github.com/sopel-irc/sopel-remind and published to PyPI as sopel-remind with an option to migrate reminders from the old core plugin via CLI
• safety
• search
• seen
• spellcheck: removed from core in 7.0 (spellcheck: remove plugin (moving to separate package) #1675); moved to https://github.com/sopel-irc/sopel-spellcheck and published to PyPI as sopel-spellcheck
• tell
• tld
• translate
• unicode_info
• units
• uptime
• url: planned (@Exirel)
• version will stay; this is a core function
• wikipedia
• wiktionary
• xkcd

Original issue text follows.

---

Assuming #1056 gets solved, it would be great to ship the core plugin set separately from Sopel itself. Doing so would make keeping plugins up to date simpler, since a broken plugin wouldn't then require an update to the whole Sopel package to fix it.

Again, this would require solving #1056 first so the core plugin reloading experience (reload.py and perhaps a couple other plugins would remain in core because they relate to administering the bot itself) remains usable. It's not acceptable to ship this before packaged plugins can be reloaded without duplicating their commands etc.

Optimistically slotted into the 7.0.0 milestone... Let's see what happens.

[dgw]

Version 7 is too soon. I was a little too optimistic about that…

Re-labeled as well. This is doable in version 8, but it might be a more gradual transition… Again, let's see what happens.

[Exirel]

After reading issues like #1435, #700, #1037, or code like the .py command, I wonder what should be a built-in module.

Here are some reasons I know (both the ones I thought about and the ones I've heard about):

• if they need configuration (like API key), they should be externals,
• if they rely on an API that is shutdown, they should be externals, or replaced promptly,
• the more complex they are, the more bug fix they might require,

Here are some advantages I can think of when doing that:

• external module can tell which version of Sopel it is compatible with,
• external module can be fixed and released without having to upgrade Sopel and all other modules,
• external module can be uninstalled without having to uninstall or modify Sopel's code,
• external module can be listed as Git Repository somewhere,
• they can have their own documentation,

Here are some issues or needs we need to address, now or later, in a near or distant future:

• "how to create a module" documentation,
• even more documentation,
• better test tools for developers of external modules,
• some documentation, too,
• a way to distribute and install single-file module without using pip install (even though pip install is the best option, not all users are able to use it properly),
• you know what? documentation,

And many more.

Oh, and while I'm here, listing things, here is how I would do the transition from "all is built-in" to "nothing is built-in":

• step 1: extract selected modules into their git repository, and their proper python package, that can be installed with a pip install sopel-module.<modname>
• step 2: in release 7.0.0, put all these ex-core modules into the setup.py's requirements,
• step 3: add a big warning message on the documentation
• step 4: remove them from the requirements of release 8.0.0

As a conclusion, I want to mention again that (we need documentation shhhh) I discuss further the needs for that to happen, rather than the if it should happen. It'll happen, but how to make sure it's a success is the question I'm interested in at the moment.

PS: I can't remember if I talked or not about documentation, which is something I want to work on myself.

[dgw]

I'm glad to see that this is already in the 8.0.0 milestone, because that ties in with the bit of discussion we had just now on IRC. The tl;dr:

• Sopel 8 will almost certainly drop Python 2 support, and that's the main thing holding us back from robust plugin reloading at this point (currently we still have some issues when things are removed, for example, or when modules exist in multiple files within a package)
• That makes Sopel 8 a great time to start this transition, because we'll likely be able to solve most or all of those issues with reloading external modules once we only have to support Python 3.x

I like @Exirel's transition plan, though the version numbers are different in my head (make most core stuff into packages for 8.0 instead of 7.0, and remove them from requirements.txt in 9.0 instead of 8.0). Changes from #1585 might alter how we choose to package stuff, but that's not really a concern for this issue—more for the PR(s) implementing it, down the road, once we've figured out what fun and useful extra stuff we can do with entry points.

[Exirel]

I like your proposal, I think we found a good middle ground. So that's cool!

However, I'd like to make an exception for the spellcheck plugin, that has been a real P.I.T.A. over the last few months. It requires system libraries: aspell, aspell-en, and libaspell-dev in order to work properly (at least in English), and it's the only one that requires that so far.

I'd like to drop it for the 7.0 release, with the proper warning in the documentation. Since the plugin system is now reliable and stable, and with options like #1585, I think it's not too of a burden on the user's end. I believe that, with the right documentation, it'll be OK.

For instance, we can suggest to follow the installation guide from the Python Aspell repository, and that the plugin itself could be installed with something close to pip install sopel-plugin-aspell.

I believe this plugin is a very peculiar one, and as such, may require an exception regarding backward compatibility.

[Exirel]

Quick update, I'm working (or planning to work on):

• help
• remind
• url

These are my first 3. sopel-help already exists and is usable out of the box, remind is close to have a first version published, and I haven't started on url yet, but it's clearly in my sight.

After that, I don't know yet.

[dgw]

I task-listed this in the OP so we can see the progress more easily. Task list behavior changes made it a bit more annoying than necessary (as outlined in community/community#4321 (comment)).

[dgw]

We've pretty clearly established that this isn't going to be an all-at-once task. Removing milestone to just let it be what the labels already say it is: A long-term planning/tracking list.

[SnoopJ]

Since #2516 is stalled on aiohttp and it looks like they're going to be in pre-release for a while, I took it on myself to write a breakout package for the ip plugin, which is the source of the aiohttp dependency. If this looks good to y'all we can bring it into the org, get the package on PyPI, and drop it from the core. Edit: done

https://github.com/sopel-irc/sopel-iplookup