Expand type hints across projects

[jaraco]

I received this inquiry in discord:

i want to put some effort into an initiative of filling jaraco.* and related projects with full-fledged type hints.
would you mind helping me by providing a short list of the most important projects that i should focus on in the first place?
thanks in advance!
my intention is to make those projects play nicely with mypy, because i find them very useful, but i also value fully typed code.

[jaraco]

I'm happy to accept contributions! I tried to find a popularity ranking of all Python packages, but I didn't succeed. I did find snyk Advisor provides a quick and easy way to find packages by first letter (https://snyk.io/advisor/packages/python/j). You could go through the list of packages there and look at the popularity of each. It also sounds as if the pypi downloads tables are available to be queried (https://pypistats.org/api/#etiquette), so you could probably construct a query to do that. You might also search Github for projects with the skeleton badge - that's a pretty good indicator that it's following my best practices.

Tips

• These projects do already run mypy checks, so are nominally typed (at least as far as mypy can infer types). Running tox will cause the type checks to be run via pytest-mypy. Running tox -- -k mypy will run mainly just the mypy tests.
• Avoid changing the implementation (in theory, hints should reflect the current, desired behavior and not require restructuring of the implementation); change the implementation only if mypy is revealing a legitimate flaw (and even then, it's better to contribute the fix in a separate commit or PR).
• I don't subscribe to email notifications on my github projects (because I'm too busy to get to them in a timely fashion), so you'll need to be patient. If you need my attention for something urgent (you're blocked or you've waited a while), it's okay to mention me to get my attention.
• I'm not a fan of type hinting noise (e.g. -> None for two-line private methods with no return). Try to focus on adding type hints that add value - that communicate something additional and non-obvious. I won't reject a PR based on this condition, but it'll be more easily accepted if it's clearly adding value and not lint.

[bswck]

I am used to type checking my code with mypy in strict mode. While some may see the strict mode as too aggressive, I will have no problem adjusting a bigger codebase to pass strict type checking. This is what I had in mind during writing the message.

I'm not a fan of type hinting noise (e.g. -> None for two-line private methods with no return). Try to focus on adding type hints that add value - that communicate something additional and non-obvious. I won't reject a PR based on this condition, but it'll be more easily accepted if it's clearly adding value and not lint.

mypy disallows untyped definitions in the strict mode. A private function without the return statement is still assumed to return Any, which is untrue, since None is the desired return type:

class Foo:
    def _bar(self):  # seen by mypy as untyped
        pass

reveal_type(Foo()._bar())  # Revealed type is "Any"

class Spam:
    def _eggs(self, biz: int):  # seen as typed, return assumed to be Any
        pass

reveal_type(Spam()._eggs(5))  # Revealed type is "Any"

Note

The only exceptions from the behavior above are __init__ and __init_subclass__ methods where -> None can be omitted, provided the function has at least one type hint in the signature.

Relevant resources:

• Allow __init__ with signature but no return type python/mypy#604,
• mypy docs: Annotating __init__ methods.

While no presence of the return statement in a method may be an obvious sign that the return type of that method does not matter, which is a logical reason not to leave a type hint there, even a return None statement with an unequivocal implication of the None return type is still seen as Any by mypy:

class Example:
    def _com(self, port: int):
        return None

reveal_type(Example()._com(43))  # Revealed type is "Any"

In places where the value of such a method is being used, the type hinting noise is then simply necessary to inform mypy about the actual return type that affects other scopes of the codebase.

I firmly believe that the best option would be being as explicit as possible when annotating types, because, as it seems, there are too few obvious cases at the end of the day.

The Zen of Python, by Tim Peters:

Beautiful is better than ugly.
Explicit is better than implicit.

_{Note: The examples were tested on mypy 1.7.0.}

[bswck]

I'm happy to accept contributions! I tried to find a popularity ranking of all Python packages, but I didn't succeed. I did find snyk Advisor provides a quick and easy way to find packages by first letter (snyk.io/advisor/packages/python/j). You could go through the list of packages there and look at the popularity of each. It also sounds as if the pypi downloads tables are available to be queried (pypistats.org/api/#etiquette), so you could probably construct a query to do that. You might also search Github for projects with the skeleton badge - that's a pretty good indicator that it's following my best practices.

Thank you for your time spent on the research. I will use these resources and compile a TO-DO list of the projects to work on in this issue. Every subsequent PR will reference this issue.

[bswck]

I firmly believe that the best option would be being as explicit as possible when annotating types, because, as it seems, there are too few obvious cases at the end of the day.

Of course if the new type hints would appear too intricate for the eye, which I can totally understand, there is always an option of creating stubs to isolate two worlds of the implementation and the type hints, like in jaraco/jaraco.functools#22. What are your thoughts on this @jaraco?

[bswck]

The roadmap is ready! 🎉

I wrote a script that extracted all jaraco projects from PyPI and sorted them by the total of downloads in the last month.
The last two projects—Distutils and backports—have 0 downloads only because checking pypi stats on them causes an HTTP 404 error. I don't think inspecting that issue is necessary.

Note

Skeleton badges come from the PyPI latest releases, not the projects' repositories.
As a result, these badges are valid indicators of the years all the relevant latest PyPI releases took place.

The script produced the following table:

	downloads last month	jaraco's role	uses skeleton?
setuptools	322655647	Owner
importlib-metadata	180078405	Owner
zipp	162258510	Owner
importlib-resources	65910632	Owner
keyring	40286684	Owner
jaraco.classes	37033262	Owner
setuptools-scm	25467595	Owner	❌
pytest-runner	16293153	Owner
configparser	10187739	Owner
jsonpickle	9339847	Owner	❌
pep517	6590096	Owner	❌
inflect	4843645	Maintainer
twine	3678588	Owner	❌
backports.functools-lru-cache	2285253	Owner
singledispatch	2269372	Owner
comtypes	1732642	Owner	❌
cssutils	1668764	Owner
jaraco.functools	1623127	Owner
path	1583434	Owner
cheroot	1482151	Owner	❌
tempora	890592	Owner
portend	829890	Owner
jaraco.collections	795281	Owner
CherryPy	790657	Owner	❌
distribute	741920	Owner	❌
jaraco.text	739296	Owner
jaraco.context	733410	Owner
keyrings.alt	614293	Owner
backports.entry-points-selectable	564770	Owner
path.py	333772	Owner	❌
Tempita	202581	Maintainer	❌
Fuzzy	120698	Owner	❌
datadiff	117694	Owner	❌
wolframalpha	51502	Owner	❌
oathtool	34623	Owner
jaraco.logging	27595	Owner
irc	22739	Owner
cherrypy-cors	18930	Owner
jaraco.stream	18120	Owner
pytest-checkdocs	14617	Owner
pytest-enabler	14499	Owner
pip-run	10838	Owner
jaraco.env	9083	Owner
jaraco.itertools	8260	Owner
hgtools	7996	Owner	❌
pytest-ignore-flaky	7441	Maintainer	❌
jaraco.path	7331	Owner
jaraco.envs	6930	Owner
backports.datetime-timestamp	6897	Owner
jaraco.versioning	6230	Owner
mongo-connector	5814	Owner	❌
pytest-perf	5729	Owner
jaraco.ui	5272	Owner	❌
jaraco.packaging	4884	Owner
jaraco.develop	4502	Owner
jaraco.structures	4416	Owner
pytest-services	4205	Maintainer	❌
rst.linker	3714	Owner
jaraco.windows	3706	Owner
jaraco.vcs	3633	Owner
backports.unittest-mock	3263	Owner
jaraco.docker	2477	Owner
suds-bis	2094	Owner
jaraco.abode	1941	Owner
jaraco.email	1704	Owner	❌
jaraco.net	1697	Owner
MagicBus	1478	Owner	❌
svg.charts	1416	Owner
jaraco.mongodb	1336	Owner
calendra	1242	Owner
jaraco.services	768	Owner
pmxbot	740	Owner
jaraco.tidelift	718	Owner
event	673	Owner	❌
jaraco.clipboard	560	Owner	❌
openpack	526	Owner
elastic2-doc-manager	504	Owner	❌
jaraco.test	499	Owner
jaraco.pmxbot	439	Owner
dynpool	312	Owner	❌
pytest-black-multipy	307	Owner	❌
cherrypy-dynpool	307	Owner	❌
jaraco.modb	269	Owner
yg.lockfile	239	Owner	❌
chucknorris	236	Owner
nspektr	225	Owner
xlsxcessive	203	Owner
cmdix	189	Owner
googlevoice	188	Owner
jaraco.financial	175	Owner
jaraco.timing	171	Owner	❌
jaraco.postgres	164	Owner	❌
jaraco.compat	163	Owner	❌
paradocx	162	Owner
http-okapi	142	Owner	❌
jaraco.geo	137	Owner
elastic-doc-manager	132	Owner	❌
jaraco.util	131	Owner
compilers	129	Owner	❌
jaraco.apt	122	Owner	❌
setuptools-svn	103	Owner	❌
jaraco.xonsh	98	Owner
vr.cli	96	Owner	❌
vr.runners	96	Owner	❌
lpaste	92	Owner
rwt	92	Owner	❌
jaraco.fabric	83	Owner
NAT-PMP	83	Owner	❌
vr.common	81	Owner	❌
pytest-home	74	Owner
fogbugz-bis	68	Owner
backports.hook-compressed	66	Owner
jaraco.nxt	61	Owner
motivation	60	Owner	❌
jaraco.video	57	Owner	❌
eggmonster	56	Owner	❌
jaraco.site	55	Owner
vr.server	53	Owner	❌
vr.imager	51	Owner	❌
backports.print_function	49	Owner	❌
librarypaste	47	Owner
backports.method_request	45	Owner	❌
excuses	44	Owner
jaraco.translate	43	Owner	❌
backports.pdb	43	Owner
jaraco.media	41	Owner
jaraco.crypto	39	Owner
pmxbot.webhooks	39	Owner	❌
popquotes	38	Owner	❌
vr.builder	37	Owner	❌
solr-doc-manager	33	Owner	❌
jaraco.input	32	Owner	❌
jaraco.home	31	Owner
jaraco.imaging	30	Owner	❌
vr.agent	29	Owner	❌
jaraco.keyring	28	Owner	❌
vr.events	28	Owner	❌
pmxbot.rss	28	Owner	❌
jaraco.xkcd	27	Owner	❌
jaraco.office	26	Owner	❌
freedompop	26	Owner	❌
treehouse	26	Owner	❌
dropbox-index	26	Owner
jaraco.zstd	24	Owner
recapturedocs	24	Owner
setuptools-hacks.bypass-summary-newline	21	Owner
pmxbot.nsfw	20	Owner	❌
vr.launch	18	Owner	❌
pmxbot.saysomething	13	Owner	❌
jaraco.parables	11	Owner	❌
yg.thumpy	11	Owner	❌
yg.eventful	7	Owner	❌
Distutils	0	Owner	❌
backports	0	Owner	❌

In order to make tracking the progress significantly easier, I've generated the following checklist-form roadmap.

Roadmap

TODO for every project linked below:

• ensure a py.typed marker is present,
• modernize type hinting,
• ensure that mypy --strict passes successfully.

---

• setuptools
• importlib-metadata
• zipp
• importlib-resources
• keyring
• jaraco.classes
• setuptools-scm
• pytest-runner
• configparser
• jsonpickle
• pep517
• inflect
• twine
• backports.functools-lru-cache
• singledispatch
• comtypes
• cssutils
• jaraco.functools
• path
• cheroot
• tempora
• portend
• jaraco.collections
• CherryPy
• distribute
• jaraco.text
• jaraco.context
• keyrings.alt
• backports.entry-points-selectable
• path.py
• Tempita
• Fuzzy
• datadiff
• wolframalpha
• oathtool
• jaraco.logging
• irc
• cherrypy-cors
• jaraco.stream
• pytest-checkdocs
• pytest-enabler
• pip-run
• jaraco.env
• jaraco.itertools
• hgtools
• pytest-ignore-flaky
• jaraco.path
• jaraco.envs
• backports.datetime-timestamp
• jaraco.versioning
• mongo-connector
• pytest-perf
• jaraco.ui
• jaraco.packaging
• jaraco.develop
• jaraco.structures
• pytest-services
• rst.linker
• jaraco.windows
• jaraco.vcs
• backports.unittest-mock
• jaraco.docker
• suds-bis
• jaraco.abode
• jaraco.email
• jaraco.net
• MagicBus
• svg.charts
• jaraco.mongodb
• calendra
• jaraco.services
• pmxbot
• jaraco.tidelift
• event
• jaraco.clipboard
• openpack
• elastic2-doc-manager
• jaraco.test
• jaraco.pmxbot
• dynpool
• pytest-black-multipy
• cherrypy-dynpool
• jaraco.modb
• yg.lockfile
• chucknorris
• nspektr
• xlsxcessive
• cmdix
• googlevoice
• jaraco.financial
• jaraco.timing
• jaraco.postgres
• jaraco.compat
• paradocx
• http-okapi
• jaraco.geo
• elastic-doc-manager
• jaraco.util
• compilers
• jaraco.apt
• setuptools-svn
• jaraco.xonsh
• vr.cli
• vr.runners
• lpaste
• rwt
• jaraco.fabric
• NAT-PMP
• vr.common
• pytest-home
• fogbugz-bis
• backports.hook-compressed
• jaraco.nxt
• motivation
• jaraco.video
• eggmonster
• jaraco.site
• vr.server
• vr.imager
• backports.print_function
• librarypaste
• backports.method_request
• excuses
• jaraco.translate
• backports.pdb
• jaraco.media
• jaraco.crypto
• pmxbot.webhooks
• popquotes
• vr.builder
• solr-doc-manager
• jaraco.input
• jaraco.home
• jaraco.imaging
• vr.agent
• jaraco.keyring
• vr.events
• pmxbot.rss
• jaraco.xkcd
• jaraco.office
• freedompop
• treehouse
• dropbox-index
• jaraco.zstd
• recapturedocs
• setuptools-hacks.bypass-summary-newline
• pmxbot.nsfw
• vr.launch
• pmxbot.saysomething
• jaraco.parables
• yg.thumpy
• yg.eventful
• Distutils
• backports

@jaraco, could you please let me know if you want to apply some additional filtering/sorting to the roadmap above?

[bswck]

An interesting approach would be to measure—for every project—how many PyPI projects depend on it, using libraries.io. But I think it would more or less correlate with downloads/month anyway, as I am assuming (without having done any research) that most of these downloads come from pipelines that install every dependency for the first time, and dependencies referencing the same project being together aren't that common (so like, when for two projects A and B sharing the same project X as a dependency, X in fact gets installed "twice less", because only once, as it is reused in the same environment by A and B, even though I would technically call X more popular in this case). This can get even more interesting if we add distinction between pip-like and pipx-like installation methods, where a shared dependency would be installed twice for separate environments of A and B, assuming A and B are CLI applications... I could be pointing out all the things that come to my mind further, and it gets very complicated and multilayered as I dig in the rabbit hole.

Yeah, so all in all I've learned that "package popularity" isn't trivial, because both the convergence in the web of dependencies as well as the statistics of total downloads in time play a role in the correct evaluation of how popular a package really is.

Anyway though, this is just an attempt to prioritize the tasks. Since some most popular jaraco projects commonly depend on some less popular jaraco projects (downloads/month-wise!), I started off from jaraco.functools and jaraco.classes.

[jaraco]

Amazing analysis. Thanks!

You'll notice that libraries.io is a project by Tidelift. You may have noticed that I also work with them as they work to garner support for open source maintainers from the enterprise users. They've probably built other tooling and may even be interested in collaborating on tooling to support open source maintenance. You may want to consider signing up with them as a maintainer (there's no cost and could potentially pay) and engaging on the forums to see if there is interest in collaborating. If you need a referral or anything to get signed up, let me know.

[bswck]

You may want to consider signing up with them as a maintainer

Thank you! I've just applied to lift a few of my projects!

---

Due to a considerable number of projects that need similar work, I created a project that aims to automate the whole process as much as possible—autorefine.

MonkeyType will turn out very handy when it comes to type hints generation. I will take care of making them as sophisticated as needed. I think all these projects have enough coverage, so I will simply generate the types by running tests.

I will leverage LibCST and create custom rules if needed for modernizing the projects (some had their last releases a few years ago)—but this is out of scope at the moment.

Contributions & suggestions very, very welcome—I am learning.

Hopefully the tool will speed up more_itertools.consume(map(functools.partial(refine, scope="typing"), jaraco_projects)).

[bswck]

I've made a Coherent OSS project for the initiative: https://github.com/orgs/coherent-oss/projects/3/views/2

[jaraco]

I firmly believe that the best option would be being as explicit as possible when annotating types, because, as it seems, there are too few obvious cases at the end of the day.

Of course if the new type hints would appear too intricate for the eye, which I can totally understand, there is always an option of creating stubs to isolate two worlds of the implementation and the type hints, like in jaraco/jaraco.functools#22. What are your thoughts on this @jaraco?

I guess that's fine. I should probably get used to Python being more verbose and less essential.

[bswck]

I firmly believe that the best option would be being as explicit as possible when annotating types, because, as it seems, there are too few obvious cases at the end of the day.

Of course if the new type hints would appear too intricate for the eye, which I can totally understand, there is always an option of creating stubs to isolate two worlds of the implementation and the type hints, like in jaraco/jaraco.functools#22. What are your thoughts on this @jaraco?

I guess that's fine. I should probably get used to Python being more verbose and less essential.

Well, I guess it's just that Python wasn't built for being statically typed. 🤷‍♀️
The good news is that due to its continuous development, the unfortunate effect of verbosity over essentialness in typing slowly decreases: take PEP 695 as an example.

[Avasam]

As more skeleton-based projects get typed (with a py.typed marker), I'd recommend requesting them to https://github.com/hauntsaninja/mypy_primer/blob/master/mypy_primer/projects.py . Especially for projects that are widely used in the ecosystem.