-
Notifications
You must be signed in to change notification settings - Fork 243
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Type stubs for single-file top-level modules #1333
Comments
I agree this is suboptimal, and I'd support lifting the restriction if we can come up with a good way to do it. Maybe @ethanhs has some insights into why we didn't provide a way to type single-module packages at the time. The first obvious solution perhaps would be to put something in the package's dist-info directory, e.g. a new key in the METADATA file. But the problem with that would be that type checkers can't reliably go from the name of the installed module to the dist-info directory, because the names may not match. |
A very simple solution could be to create a I am unfamiliar with the process of amending a PEP, are there other places where I should announce this discussion? PEP 561 has been marked as "Final", does this mean that introducing such an update would require a new PEP? |
This would technically require a new PEP, yes. However, it can be short. |
I think just |
Good point, Shantanu! Oh ... I just realized something ... static type checkers could just as well check for Ok thanks Jelle, I'm working on a PEP draft for this right now. |
I think |
Right I don't really consider having to change
So the drawbacks I described that exist for shared libraries pretty much don't apply to pure Python modules. I don't think we should introduce yet another type of I think we should rather go with the intuitive solution of putting a |
I agree this is the most straightforward approach for single-file binary (compiled) packages. Pyright already supports a ".pyi" file in site-packages if it's present. It looks like mypy doesn't handle this currently, but I'm guessing it would be a simple change. |
I wrote a draft PEP "Type stubs for single-file top-level modules". Feedback is very much welcome :) (This is my first attempt at writing a PEP.) |
|
I'm supportive of the idea and I'm happy to co-sponsor :) However, I wouldn't want to be the sole sponsor — I haven't written or sponsored a PEP before, so I'm not 100% sure of the exact process. I also feel like I'm pretty close to full capacity on my open-source commitments at the moment. |
I've seen this in a few places; I'll also see if I can dig up some references. |
Thanks for putting the effort into writing this up (and starting this discussion in the first place)! Given that PEP 561 95% solves this problem, I feel if we want to make changes to standards here, we shouldn't solve only half of the remaining 5% of the problem. I'm not sympathetic to the claim that pure Python developers don't feel the drawbacks you mention, mainly because I think 2/3 of those drawbacks are very weak: "you can import my_project._native" (so what? consenting adults), "potentially misleading developers into thinking the package is implemented in Python" (such developers would also believe that numpy is in pure python)... ...I think the biggest reason to do this is just "it's annoying to complicate project layout because of a dumb shortcoming of type checkers" and "why have package when you can have module, simple is better than complex". This applies equally to single pure Python modules with inline types and single extension modules. |
^I agree with everything @hauntsaninja just said; I also think it would be a real shame to not find a way to solve this for pure-Python file packages |
The suggested approach of putting a |
Yes, but not for inline types, which I strongly encourage as the most maintainable way to add types to pure Python |
Ah right, good point. Possible hacky solution: Use a .pyi file, but put some special marker code in the .pyi file that indicates "look inline". |
So, if type checkers see a |
Thanks everybody :)
There is #1297.
Yes I agree with that.
If both
@hauntsaninja I agree with 2/3 drawbacks being weak. I think the main drawback is that tooling such as documentation generators generally don't have special support for recognizing such re-exporting I am alright with also solving the problem for pure-Python modules with inline types. My initial idea just now was to create a I think I'd rather introduce a new file extension for new marker files (e.g.
What do you think? Perhaps the |
There is already a way to handle inlined types for a single-file module: convert it to a multi-file package and add a "py.typed" marker. This isn't too onerous. Let's not add hacky and inconsistent solutions (like symbolic links or files with specific name extensions). Resist the urge to overreact to one or two people grumbling about needing to do a few extra steps (one time) to make this work. There is currently no way to package type information for a single-file compiled module. The side-by-side ".pyi" file is an elegant solution to this currently-unsolved problem. Let's focus on solving this problem, not creating more complexity and inconsistencies to solve a problem that already has a solution. |
I think a key difference between pure modules and extension modules that has not been mentioned here yet is that the I have strongly revised my PEP draft to better explain the reasoning (as well as answering the questions raised by @JelleZijlstra). Sidenote: I have now also specified that these top-level @hauntsaninja There are three scenarios:
Since the first two cases always need a Addressing the third case would require a solution that is not obvious (defining magic variables like |
True, however I've seen project owners rejecting adding the type hint marker because it would have required them to move to a package structure. It would be a much easier sell, if there would be a solution which didn't require a structure change. I do agree that adding a separate file with a new suffix would be too complicated. What about adding |
Really? That surprises me. Can you provide examples? I guess I'm not very sympathetic to this argument. This is a really low bar. It involves a one-time change that requires just a few minutes of work. If a library maintainer is unwilling to do this, then they're just looking for excuses not to support typing. I don't think that inventing redundant mechanisms is the right solution to this problem. |
Unfortunately not. It was some time ago and IIRC the owner wasn't too convinced about the usefulness of typing.
The refactoring might be simple but packaging is another story. Sometimes it's easier to leave it alone if it works. -- |
Hi! Sorry for not responding sooner, been a bit busy recently. Let me start by giving context into why PEP 561 makes the tradeoffs it does, and what I was discussing with type checker authors at the time.
That being said, I do think PEP 561 is a bit lacking.
I've actually also seen this, but I cannot recall where. I do still think the bar is pretty low, but I also think the UX for marking a package as typed could be better. One of the original alternate designs for PEP 561 was to include typing support status in the distribution metadata. This would most likely exist as a list of files in the distribution that support typing or something like that. This solution is particularly appealing now that 3.7 is almost end of life (June of this year), and so soon all versions of Python will support I agree though that if we don't want to shift to something like the above metadata-based system, keeping the status quo and suggesting maintainers change the layout of the package is an acceptable solution. |
I'm very likely missing something well known here, but could someone tell me why a type checker needs a marker to consider looking inline rather than just looking inline unconditionally? |
@Kentzo it's a good question, particularly in 2023. The main reason is that if a package doesn't have types or only has partial types, it's useful to warn a user about that so they don't falsely think they have typing coverage. This also gives the user the opportunity to install a stubs package themselves / the type checker to easily detect this situation and hint to do so. Historical reasons are that annotations weren't always reserved for typing use and type checkers sometimes struggled with unusual code. If I were writing a type checker in 2023, I'd probably always try to analyse the code because more information is usually better and you can still type check the shape of things, but I'd use the absence of py.typed to surface an error to the user. |
FYI I ran into this recently due to the lack of support for top-level |
i dont like it, but that's how it'll have to be python/typing#1333 s+: also, add __str__ support for query types (#18)
This required moving away from using a single module, a known limitation being currently visited in python/typing#1333.
### Summary Mypy only considers the type annotations of packages that contain the `py.typed` marker file, even if they are fully type-annotated (like `pybreaker`). This PR adds said file. I had to move `pybreaker` into a module instead of a single .py file, because there is no support for `py.typed` files for single .py file packages yet (see python/typing#1333). ### Test Plan - Ran `python setup.py test` to ensure the tests still pass. - Ran `python -m build` to build the package. - Observe the output logs to see that `py.typed` was included. - Install the tarball into a project where I have mypy set up. Run mypy and observe mypy now no longer fails with `... becomes "Any" due to an unfollowed import [error/no-any-unimported]` for imports from `pybreaker`.
### Summary Mypy only considers the type annotations of packages that contain the `py.typed` marker file, even if they are fully type-annotated (like `pybreaker`). This PR adds said file. I had to move `pybreaker` into a module instead of a single .py file, because there is no support for `py.typed` files for single .py file packages yet (see python/typing#1333). ### Test Plan - Ran `python setup.py test` to ensure the tests still pass. - Ran `python -m build` to build the package. - Observe the output logs to see that `py.typed` was included. - Install the tarball into a project where I have mypy set up. Run mypy and observe mypy now no longer fails with `... becomes "Any" due to an unfollowed import [error/no-any-unimported]` for imports from `pybreaker`.
This is necessary in order for type checking due to: python/typing#1333 Given that _ldap is an internal module, this change is hopefully ok.
Workaround-For: PyO3/maturin#1399 Workaround-For: python/typing#1333
Unless I'm wrong you can have (single file) modules or (multi file) packages. Again, unless I am wrong, modules are not deprecated. As long as modules are a valid way to produce libraries, I think it is fair to demand a way to indicate they're typed without having to change them. |
PEP 561 currently states the following:
I consider this requirement to be problematic for Python libraries that are written in another programing language and distributed as compiled
.so
files. PEP 561 currently does not provide a way to mark.so
files residing directly insite-packages/
to be typed, resulting in typed shared libraries needing to introduce an intermediary__init__.py
file such as the following:While this works for static type checkers I think this is obviously suboptimal because it has several undesired side-effects. Let's take the following as an example:
The unintended side-effects are:
my_project._native
._native
shows up in the documentation generated by pydoc. E.g. under PACKAGE CONTENTS for the documentation ofmy_project
and invoking e.g.help(my_project.foobar)
will tell you thatfoobar
resides in the modulemy_project._native
.my_project.__file__
now is the__init__.py
file instead of the.so
file, potentially misleading developers into thinking the package is implemented in PythonSo I really think PEP 561 should be amended to provide some way of marking single-file packages as "typed" without having to resort to hacks such as defining an intermediary
__init__.py
since that introduces a bunch of undesired side-effects that have the potential to confuse API users.What do you think about this?
The text was updated successfully, but these errors were encountered: