7.0.0

Changes between 6.6.9 and 7.0.0

Sopel 7.0 contains numerous fixes, tweaks, new features, and internal
improvements. While we maintain no official statistics on such things, it's
probably the biggest release the project has ever seen.

The full, detailed list of changes is below, but you can just read the
migration guide if all you care about is the really
important stuff. All breaking changes (including those planned for the next
release) are explained there.

Plugin changes

• The spellcheck plugin has been removed to simplify dependencies [#1675]
  • An updated version with added features and tweaks, based on aspell, is
    installable separately from PyPI [#1164, #1545]
  • See #1142 & #1642 for background on why we decided to separate
    spellcheck into a separate package (tl;dr: dependency hell)
• Similarly, the ipython plugin is now an external package, eliminating from
  Sopel itself a dependency which most users will never need [#1684]
  • Find the new sopel-ipython package on PyPI
• The .choose/.choice command has been moved from dice into a standalone
  plugin, aptly named choose [#1679]
• Moved .py into its own plugin, named py [#1710, #1711, #1712]
  • In case of issues with our "official" instance, there's now a setting to
    configure the address of your own Oblique service
• The currency plugin changed data sources to support more currencies, and
  can optionally use Fixer.io for even more [#1430, #1627, #1629]
  • Fixer.io requires a free API key, which gives plenty of calls per month
    (the plugin caches exchange rates for 24 hours)
• Choose any of five pastebin services for help output [#1451, #1651]
  • This is intended mostly for resilience (so one pastebin service going
    down, as ptpb did during the 6.6.x life-cycle, won't force a new Sopel
    release), but it's also just good to have choices
• Optionally hide IRC server name/address in help command listing [#1459]
• New help plugin setting, reply_method [#1700]
• wiktionary now supports many more parts of speech [#1443]
  • Definitions can be retrieved for things like proper nouns, prepositional
    phrases, and punctuation marks—things that were unsupported in Sopel 6.x
  • This means that the wiktionary plugin is now somewhat case-sensitive, to
    account for common and proper nouns that differ only in capitalization
• url will now ignore "private" addresses by default [#1439, #1624]
  • New config settings for the url plugin allow overriding the checks, in
    cases where loading previews of e.g. LAN servers is safe
  • The default is off, so users don't unwittingly open themselves to attackers
    fishing for running HTTP services on the local machine or network
• url now correctly ignores invalid URLs (e.g. http://*\.com) [#1788]
• Various improvements to the clock functions [#1592]
  • Improved guessing in .t/.time command, which no longer falls back all
    the way to the bot's default timezone if given an unrecognized argument
  • New .tz command, to explicitly get time for a timezone name (in case of
    conflict between a known nick and a timezone name)
• The tell & remind plugins' ".db" files have changed names [#1699]
  • Both will attempt automatic migration of existing files, if they exist,
    and output debugging information if the migration fails
• The .at command (in remind) understands dates now [#1590, #1736]
  • Finally, it's possible to set a reminder more than 24 hours ahead without
    first converting the future date/time to a duration for use with .in!
• A new tell plugin setting allows delivering messages privately [#1694]
• The reddit plugin now also handles short redd.it links, direct links to
  comments, inline u/ and r/ references, & reddit-hosted image/video links
  [#1503, #1720, #1722, #1734, #1760, #1773]
• Spoilers & NSFW are now separate concepts in the reddit plugin [#1620]
  • Reddit implemented spoilers as a distinct post flag some time ago. Sopel's
    plugin supports independently setting channels as "SFW" or "spoiler-free".
• Rolling .dice now officially supports trailing # comments [#1577]
• Python version is now included in .version command output [#1462]
• The .version command's output format is improved [#1633]
• Using the .reload command shows the specific file reloaded [#1762]
• The .seen command's output now uses relative time [#1661]
• Readability of .choose command output is significantly improved [#1425]
• Added .invite command [#1497]
  • Both Sopel and the inviting user must have privileges in the target channel
• A .restart command is added [#1333]
  • Usable by admins only, just like .quit
• The .msg command is now known as .say [#1606]
  • .msg will continue to work for now, likely until being removed in Sopel 8
• The admin plugin auto-saves channels when using .join/.part, and added
  .tmpjoin/.tmppart commands to bypass this behavior [#1492]
• Added command to .unset config values in the admin plugin [#1556]
• Restored commands in adminchannel plugin for managing op/voice [#1498]
  • These were removed some time ago, seemingly without reason, by the
    project's previous maintainers
  • Since there was some desire from users to have them back, we restored them
• Fixed/tweaked hostmask handling in adminchannel ban functions [#1791]
• find also collects Sopel's own messages now, so users can "correct" the bot
  if they wish to be extra cheeky [#1470]
• Fixed that the url plugin had to be enabled or some link-handling functions
  wouldn't work [#1510]
• Tweaked .ddg command output so it's less likely to mangle URLs [#1713]
• remind commands now "reply" with error messages [#1715]
• Fixed etymology plugin error with empty argument [#1677]
• Fixed an uncaught exception in instagram plugin [#1702]
• Handle JSON fetch/parse errors in find_updates plugin [#1779]
• Removed nonsensical uses of the core.verify_ssl setting [#1706]
• Unused clock plugin settings have been removed [#1696]
• Updated MaxMind database handling in ip plugin [#1797]
• Reworked safety plugin's cache management [#1753, #1802]
• General code cleanup and tweaks all around [#1402, #1486, #1505,
  #1569, #1573, #1578, #1579, #1581, #1592, #1606,
  #1607, #1609, #1678, #1681, #1696, #1717, #1721,
  #1725, #1735, #1741, #1754]

Core changes

• Brought back support for non-SQLite databases by switching to SQLAlchemy
  [#1446, #1652, #1729, #1755, #1774, #1777, #1783]
  • For details on using non-SQLite databases, see the README or
    configuration instructions
  • You will probably need to install additional dependencies if you wish to
    use something other than SQLite
  • Migrating an existing database from SQLite to your chosen option is
    probably easy, but we do not offer instructions for doing so
  • Be aware that plugins written for older versions of Sopel might not work
    properly with non-SQLite databases
• The db_filename config setting (for SQLite) is now interpreted as relative
  to the config's homedir setting [#1574]
  • homedir itself has a default value that will be used if not set
• Added separate server & nickname authentication options [#1513]
  • See authentication configuration docs
• Added commands_on_connect setting to send a list of commands automatically
  when Sopel's IRC connection is successfully established [#1528]
• Log files have become much more configurable [#1678, #1714]
  • Logs are named based on the config name
  • Many, many new settings added to customize logging
  • The default log format includes timestamp, source package, & level (which
    will make logs much more useful when reporting bugs)
• Logs now have information about which plugin file was reloaded [#1762]
  • This is helpful for owners of Sopel instances with multiple versions of a
    plugin available for testing or development purposes
• Sopel's own rate-limiting & flood protection parameters are now configurable,
  and can even be turned off entirely if Sopel is behind a bouncer or other IRC
  proxy that handles flood protection itself [#1518, #1638]
• Added more control over JOIN throttling [#1751]
  • Includes a new throttle_wait setting instead of a hard-coded time value
• Log timestamp and log line formats are now configurable [#1512]
  • See details in the logging configuration docs
• Log filenames now include the config name, to help keep track of logs from
  multiple Sopel instances [#1547]
• Home directory is no longer assumed to be ~/.sopel on first run [#1404]
• Restarting Sopel via CLI is added [#1333]
• Sopel's CLI is restructured [#1493, #1509, #1718]
  • New subcommands (start, stop, restart, and configure) replace many
    of the old --options, cleaning up the syntax
  • The legacy --options will continue to work for the life of Sopel 7.x, and
    will be removed in Sopel 8
• New --config-dir common option [#1598]
• Added a new sopel-config command for working with config files [#1507]
  • Currently supports list (existing files), init (new config file), and
    get (config value) actions
  • The old --list argument to sopel is considered deprecated, and will be
    removed in Sopel 8
  • See detailed usage in your terminal with sopel-config --help, or review
    the online CLI docs at our website
• Added a new sopel-plugins command for managing plugins [#1588]
  • Currently supports list (available plugins), show (plugin details &
    status), enable & disable (edits config on the user's behalf)
  • See detailed usage in your terminal with sopel-plugins --help, or review
    the online CLI docs at our website
• Sopel now also looks for plugins in $HOMEDIR/plugins [#1747]
  • This is part of a longer-term plan to reduce confusion over the term
    "module", which Sopel has been using in conflicting ways; see [#1738]
• The config file Sopel should use can be specified via the SOPEL_CONFIG
  environment variable [#1473]
• List values in config can be separated by newlines [#1628, #1690]
  • Newline-separated values support commas within each value
  • Comma-separated value support will end someday, but likely not till Sopel 9
  • Find more details in the ListAttribute documentation
• Config options can be set/overridden via environment variables [#1096]
  • Variable naming format: SOPEL_SECTION_OPTION
  • Underscores in SECTION and OPTION names are preserved, e.g.
    SOPEL_CORE_AUTH_PASSWORD
• Example multi-instance systemd template is now available [#1059]
• Example systemd unit files wait until networking is connected before starting
  Sopel [#1511]
• Running Sopel on an unknown OS platform will output a warning encouraging the
  user to report any issues (because test coverage on an unrecognized platform
  name is likely to be nil) [#1487]
• Cleaned up or refactored a bunch of places [#1424, #1429, #1456,
  #1458, #1472, #1479, #1510, #1522, #1527, #1542,
  #1557, #1561, #1567, #1579, #1580, #1583, #1597,
  #1610, #1635, #1685, #1697, #1708, #1716, #1723,
  #1724, #1728, #1730, #1731, #1732, #1735, #1739,
  #1740, #1741, #1742, #1743, #1754, #1759, #1787]
• Sopel 7 will emit warnings when run under Python 2, as Python 2.7 support
  officially ended on January 1, 2020 [#1488, #1795, #1800]
  • Sopel's warnings are set to become more dire around the time when Python's
    maintainers plan to release the final version of 2.7
  • While Sopel 7 is intended to remain compatible with Python 2, future
    compatibility is not guaranteed, and users should plan for Sopel 8 to
    officially drop Python 2 support (see upgrade notes)
• Added support for IRCv3 echo-message capability, which Sopel will now
  request upon connecting to an IRC server [#1470, #1672, #1674]
• Added support for disabling commands (or entire plugins) on a per-channel
  basis [#1235]
  • See how it works: Per-channel configuration
• Fixed tracking user away state [#1663, #1664, #1666, #1703]
• User information is now periodically updated [#1664]
• Fixed a case in which MODE tracking could break [#1737]
• Handle non-standard +y/+Y OPER modes [#1671]
  • A new OPER constant in sopel.module now exists for these channel modes,
    which (at least on InspIRCd) use the privilege prefix !
  • This is a stopgap for one specific case; we are working on further changes
    to support dynamic parsing of privilege modes/prefixes at connect time
• Stopped sending TOPIC command on JOIN [#1749]
  • IRC servers should send the topic on join without being asked, per spec
• Greatly improved the warning printed when a deprecated function is used,
  adding detail and removing unnecessary traceback lines [#1568, #1613]
  • Typical length of output for each deprecated function call reduced from
    roughly 15 lines to 3 (warning, file/line, and offending code snippet)
  • Added optional @deprecated decorator arguments:
    • reason: why this item was deprecated
    • version: the version in which the deprecation happened
    • removed_in: the version in which the deprecated item will be removed
• The end-of-life warning for users on Python 2.x is now date-aware [#1756]
• Made sure ignored users cannot trigger URL handlers [#1806]
• Tightened up some more dependency version specifiers [#1807]

API changes

• API documentation has been almost entirely overhauled [#1563,
  #1566, #1646, #1668, #1669, #1680, #1719, #1727,
  #1735, #1750, #1766, #1770, #1771, #1772, #1775,
  #1776, #1778, #1782, #1813]
  • Nearly every file, both for the public API and Sopel's internals, was
    reviewed in its entirety to make these improvements globally:
    • More consistent style
    • Better use of Sphinx features (e.g. parameter definitions, return value
      notes, & version annotations)
    • General cleanup and copy-editing
    • Add more detail and examples to help new bot users and plugin authors
    • Remove or correct outdated information left over from previous versions
• Most of Sopel's submodules now define __all__, limiting namespace pollution
  from using * in imports [#1582, #1727]
• Plugins can register themselves via setuptools entry points [#1585]
  • This feature is an evolution of the previous mechanism to install plugins
    via PyPI packages, which required a specific directory structure and
    package name format (sopel_modules.plugin_name)
  • See the entry point plugin documentation for more
• Logging has been reworked [#1678]
  • The sopel.logger.get_logger() function is deprecated in favor of a new
    sopel.tools.get_logger() utility with plugin-specific behavior
  • sopel.logger.get_logger() will begin emitting deprecation warnings in
    version 8.0, and will be removed in 9.0
• Testing tools have been overhauled [#1731, #1732, #1781]
  • The old "Mock" classes in sopel.test_tools are now deprecated:
    • MockConfig
    • MockSopel
    • MockSopelWrapper
  • New pytest fixtures make the real objects usable in tests directly
  • The bot keeps track of running triggered threads, so tests can be sure that
    processing has finished before evaluating results
  • Sopel also now exports a pytest plugin, for convenience
• module.event() decorator no longer requires a rule [#1693, #1709]
  • Since 99% (unscientific guesstimate) of event decorators were paired with
    @module.rule('.*'), that's now implied if no rule decorator is present
• Added sopel.tools.web (replaces sopel.web) [#1616, #1670]
  • Both old and new import locations will work until Sopel 7 end-of-life
  • The sopel.web package will be removed completely in Sopel 8
  • Functions marked as deprecated in the old location (e.g. web.get()) do
    not carry forward to the new package namespace; they remain deprecated
• Added tools.web.unquote(), the reverse of tools.web.quote() [#1681]
  • Note: This is not available in the sopel.web compatibility layer
• Added a set of methods in the bot object to manipulate URL callbacks:
  bot.register_url_callback, bot.unregister_url_callback, and
  bot.search_url_callbacks [#1508, #1808]
  • Modules should switch to using these new API methods instead of directly
    accessing bot.memory['url_callbacks']
  • There are no definite plans to remove bot.memory['url_callbacks'], but it
    should be considered deprecated
• Added kick() method to bot object [#1539]
  • bot.kick(nick, channel, optional_message) is shorthand for the pattern
    bot.write(['KICK', channel, nick], optional_message), used by a number
    of both core and third-party plugins
• Added bot.myinfo, containing the server's RPL_MYINFO (004) data [#1769]
  • More information in the MYINFO documentation
• Added bot.isupport, exposing network-specific settings and properties from
  the server's RPL_ISUPPORT (005) data [#1758]
  • This includes useful things like the network's maximum nickname length
    (NICKLEN), maximum topic length (TOPICLEN), number of targets allowed
    per command (TARGMAX), and channel-privilege mappings (PREFIX)
  • More information in the ISUPPORT documentation
• Added session() method to bot.db [#1811]
  • db.connect() now logs a message when used with a non-SQLite database
    connected, as raw connection behavior for different DB types varies
• Added delete_{nick,channel}_value methods in bot.db [#1526]
• Added "plugin value" API to bot.db [#1621]
  • Functionally identical to "nick" and "channel" values, but in a separate
    namespace intended for plugins' use to store key-value data that isn't
    associated with a nick or channel and doesn't belong in the config file
• Added optional default kwarg to db.get_*_value() functions [#1673]
  • This allows streamlining some uses of values fetched from the database
• New sopel.module.output_prefix decorator [#1701]
  • Defines a prefix for all output sent from the decorated callable via
    bot.say or bot.notice (bot.action & bot.reply don't fit the use
    cases this feature is intended to address)
• New sopel.module.require_account decorator [#1733]
  • This is useful to require services login on networks where Sopel can track
    users' authentication status, but will block all use of the decorated
    callable on networks without the necessary features
• sopel.module decorators work much more consistently [#1632]
  • All relevant decorators accept multiple arguments at once, and will work
    properly if used multiple times on the same function
• Added reply option to module.require_* decorators [#1456, #1500]
  • Passing reply=True will send the error message as if via bot.reply(),
    prefixing it with the name of the calling user to get their attention
  • Passing reply=False (or omitting the argument) will behave exactly as
    before, with the error message sent as if via bot.say()
• Fixed the @module.url decorator to work always, regardless of whether any
  core plugins are enabled [#1510, #1576]
• Added bot.hostmask property (read-only) [#1537]
  • This property is a shortcut for bot.users.get(bot.nick).hostmask, so it
    will throw an error if the bot is not connected to a network and joined
    to at least one channel. Basically, don't try to use it in plugin setup()
    or shutdown() methods.
• Added bot.get_plugin_meta() method to fetch plugin information [#1762]
• Added module.echo decorator [#1470]
  • Decorated callables will receive messages that Sopel itself sends,
    regardless of whether the IRC server actually supports echo-message
  • Don't send output from @echo-decorated callables; this will cause loops
• Added module.action_commands decorator [#1660]
  • This presently conflicts with other command/trigger decorators because of
    internal implementation details, but fixing that is on our to-do list
  • If you want both action_commands and another command type (commands,
    nickname_commands, etc.), use @action_commands('foo') to decorate a
    wrapper function that simply calls the main one
• Officially deprecated the methods SopelMemory.contains() and
  SopelMemoryWithDefault.contains(), to be removed in Sopel 8 [#1563]
  • Use the in operator instead
• Officially deprecated the bot.msg() method [#1606]
  • Adds a warning to Sopel's output for any third-party code that is still
    using bot.msg() (which was declared deprecated and removed from the API
    docs in 6.0) instead of bot.say()
• Added a new user_help kwarg to @module.example decorator [#1403]
  • This allows multiple help examples per command, as requested in #1200
  • See the Sopel 7 migration guide for details on using
    this new parameter, and about backwards compatibility
• Added a new online kwarg to @module.example decorator [#1555]
  • Example tests that require an Internet connection may be marked with
    online=True to indicate this fact (the default is False)
  • This facilitates running only offline-safe plugin tests, by passing
    --offline to pytest, if an Internet connection is unavailable
  • Outside of the test suite, this parameter has no effect
• Corrected case-mapping in tools.Identifier class [#1744]
  • See the Sopel 7 migration guide for details,
    particularly if your plugin interacts with the database and/or user/channel
    identifiers in their lowercase form
• Added more utilities to tools.time:
  • tools.time.seconds_to_human(), for generating human-readable fuzzy
    relative timestamps from timedelta or raw number of seconds [#1560]
  • tools.time.get_nick_timezone() and tools.time.get_channel_timezone(),
    for cases where the fallback behavior(s) of tools.time.get_timezone()
    would be undesired [#1592]
• Made tools.time.validate_timezone() more robust [#1707]
• Added alias for 462 numeric in tools.events [#1665]
  • RFC 2812 defined an incorrect spelling, ERR_ALREADYREGISTRED, and
    Sopel copied it verbatim
  • The spelling ERR_ALREADYREGISTERED can be used now, too
• Fixed invalid raw lines in generated @example tests [#1499]
• Fixed PreTrigger objects missing the text attribute sometimes [#1782]