Automated versioning for releases

[luizirber]

#2393 broke the use of tags for auto-versioning (as supported by setuptools_scm), and turns out that setuptools_scm is doing something that PyPA doesn't recommend anymore. From core metadata and PEP 643:

The fields Name and Version MUST NOT be marked as Dynamic

There are a couple of approaches to fix that, in order of complexity:

• Manual update of versions before releases. Need an extra PR (for bumping version in pyproject.toml), and then tag the merged PR in latest. Can be automated a bit with bump2version or similar packages. Tox has a special target and script to further automate the process.

  • PROS: simple, tooling is mostly under our control.
  • CONS: higher chance of mistakes during release process. Still need to write changelog manually (like we do now).

• Conventional commit automation, like release-please, example from rust-bio here with conventional commits checker here. This combination is nice because changing the PR title is enough to satisfy checks (and generate the changelog).

  • PROS: auto-generate changelog, standing PR for new version (just need to merge it).
  • CONS: more machinery (and maintenance when it breaks), incompatible with current (manual) changelog approach

Aiming to fix it by doing the first one (updating release docs, probably in #2386?), while exploring the second one in another repo/fork.

[ctb]

hey @luizirber it might be time for a new release sometime soon - can I do a release manually, do you think? or can you point me in the direction you're thinking of for automated releases, so I can work on that in my spare moments?

[luizirber]

hey @luizirber it might be time for a new release sometime soon - can I do a release manually, do you think? or can you point me in the direction you're thinking of for automated releases, so I can work on that in my spare moments?

Main change to current release process is (maybe?) creating a PR for the version bump, and tag -rc versions from it. From a quick look at #2386 this would be:

• Testing step 1
  • create a new branch for the release PR
  • bump version in this PR
• Testing step 3
  • Many fixes on deps here =(
• How to make a final release step 0
  • merge the PR, and then proceed with step 1 (tagging from latest)

This mostly fits the "manual" approach to releases. For the conventional commits/automated I don't have any better pointers than the ones in the issue description 😅

[luizirber]

I updated PR #2386 with the changes from #2393 and implementing a manual version bump.

Maturin is stricter on what is added to the source dist (and wheel), so the list from 4.6.1 is much smaller now, but also missing some interesting files (like CITATION.cff).

I added some back here, but there might be others we want too? (will probably open a separate PR for pyproject.toml updates, and leave #2386 only for docs -> added to #2432)

[ctb]

Hey @luizirber the fun continues!

I'm tracking some errors in our packages that use sourmash that seem to come from this code in sourmash:

from pkg_resources import get_distribution, DistributionNotFound

try:
    VERSION = get_distribution(__name__).version
except DistributionNotFound:  # pragma: no cover                                
    try:
        from .version import version as VERSION  # noqa                         
    except ImportError:  # pragma: no cover                                     
        raise ImportError(
            "Failed to find (autogenerated) version.py. "
            "This might be because you are installing from GitHub's tarballs, "
            "use the PyPI ones."
        )

which, I think, is no longer needed after <gestures vaguely at #2386 #2342&rt;.

I'll start a PR to remove them and see what breaks! 🎉

errors in other packages

over in genome-grist, I'm getting:

DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('mpl_toolkits')`.
Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages

which is turned into an error because I'm using python -Werror. Working on handling that separately ;). But it brought the above pkg_resources code to my attention...

[ctb]

#2491 is passing tests. Will fix up PR description and submit for merge at some indefinite point in the future ;).

[luizirber]

I'm tracking some errors in our packages that use sourmash that seem to come from this code in sourmash:

yup, definitely should be changed... I fixed it in screed but not in sourmash.
https://github.com/dib-lab/screed/blob/ba1d4ebbdbe3dbfdea702c6e01322e9f62c12193/screed/__init__.py#L39L51
it is still longer because it has the setuptools_scm fallbacks, but we want to use importlib.metadata and not pkg_resources (which depends on setuptools)

The catch is that it is really hard to get a venv WITHOUT setuptools installed (because some dependency might bring it in) to check that we are not depending on it...