Handling a function with the contextlib.contextmanager decorator that does not implement __enter__ or __exit__ ?

[HunterAP23]

Working with the psutil library, they have a Process class with the method oneshot that is meant to retrieve information on the system. Here is the definition of that function.

To improve performance, they grab a threading.RLock() variable named self._lock, then decide whether to use cached information or to grab the new information. Since psutil was built to work on multiple operating systems (Windows, Linux, MacOS), they do not actually implement the __enter__ and __exit__ methods that the @contextlib.contextmanager decorator provides - they instead call predefined oneshot_enter and oneshot_exit functions defined in each of the different platform code they have written.

This leads to issues where utilizing the psutil.Process().oneshot() method with the with keyword gives an error. For example:

import psutil

my_process = psutil.Process() # gets the process with the current PID
with my_process.oneshot():
    print(my_process.pid) # just print the PID 

Scanning this with tools like pyright will result in the following errors:

error: Object of type "None" cannot be used with "with" because it does not implement __enter__
...
error: Object of type "None" cannot be used with "with" because it does not implement __exit__

I'm wondering if this is something that needs to be fixed within the typing / typeshed libraries, or if psutil would need to change the psutil.Process().oneshot() method.

[hauntsaninja]

The typeshed stubs are incorrect: https://github.com/python/typeshed/blob/c0030b2b12738aefbbe71b857c20776b09d34e64/stubs/psutil/psutil/__init__.pyi#L90
PR welcome!

[HunterAP23]

Ah perfect, going to get started on it.

[HunterAP23]

So then the question is deducing what the return types are
Looking at the source code from psutil, the oneshot function either yields nothing, OR if an error occurs then it will raise an AttributeError if the method cache_deactivate that oneshot calls has an error. Here's the definition for cache_deactivate:
https://github.com/giampaolo/psutil/blob/12994a739dbeae7a4e7273cb742ca9270e168763/psutil/_common.py#L462-L467

I'm not sure how to handle a contextmanager's return values when it either yields None or raises an exception. The CONTRIBUTING.md file doesn't really explain it, or at least it does not work in this case as I still get both errors I mentioned above.

[JelleZijlstra]

Exceptions aren't reflected in the types. I'd need to look at this code more to be certain but it sounds to me like the function should simply return ContextManager[None].

[sobolevn]

I've opened a PR with the fix: python/typeshed#6006

[HunterAP23]

Semi related question, but what is the reason for doing import X as X when the name is not aliased to anything different?
psutil has some chunks like this that don't really make sense to do for me, liek this one:

from ._common import (
    AIX as AIX,
    BSD as BSD,
    CONN_CLOSE as CONN_CLOSE,
    CONN_CLOSE_WAIT as CONN_CLOSE_WAIT,
    CONN_CLOSING as CONN_CLOSING,
    CONN_ESTABLISHED as CONN_ESTABLISHED,
    CONN_FIN_WAIT1 as CONN_FIN_WAIT1,
    CONN_FIN_WAIT2 as CONN_FIN_WAIT2,
    CONN_LAST_ACK as CONN_LAST_ACK,
    CONN_LISTEN as CONN_LISTEN,
    CONN_NONE as CONN_NONE,
    CONN_SYN_RECV as CONN_SYN_RECV,
    CONN_SYN_SENT as CONN_SYN_SENT,
    CONN_TIME_WAIT as CONN_TIME_WAIT,
    FREEBSD as FREEBSD,
    LINUX as LINUX,
    MACOS as MACOS,
    NETBSD as NETBSD,
    NIC_DUPLEX_FULL as NIC_DUPLEX_FULL,
    NIC_DUPLEX_HALF as NIC_DUPLEX_HALF,
    NIC_DUPLEX_UNKNOWN as NIC_DUPLEX_UNKNOWN,
    OPENBSD as OPENBSD,
    OSX as OSX,
    POSIX as POSIX,
    POWER_TIME_UNKNOWN as POWER_TIME_UNKNOWN,
    POWER_TIME_UNLIMITED as POWER_TIME_UNLIMITED,
    STATUS_DEAD as STATUS_DEAD,
    STATUS_DISK_SLEEP as STATUS_DISK_SLEEP,
    STATUS_IDLE as STATUS_IDLE,
    STATUS_LOCKED as STATUS_LOCKED,
    STATUS_PARKED as STATUS_PARKED,
    STATUS_RUNNING as STATUS_RUNNING,
    STATUS_SLEEPING as STATUS_SLEEPING,
    STATUS_STOPPED as STATUS_STOPPED,
    STATUS_TRACING_STOP as STATUS_TRACING_STOP,
    STATUS_WAITING as STATUS_WAITING,
    STATUS_WAKING as STATUS_WAKING,
    STATUS_ZOMBIE as STATUS_ZOMBIE,
    SUNOS as SUNOS,
    WINDOWS as WINDOWS,
    AccessDenied as AccessDenied,
    Error as Error,
    NoSuchProcess as NoSuchProcess,
    TimeoutExpired as TimeoutExpired,
    ZombieProcess as ZombieProcess,
)

The only explanation I could find is in the CONTRIBUTING.md file where it says:

Imports in stubs are considered private (not part of the exported API)
unless:
* they use the form ``from library import name as name`` (sic, using
  explicit ``as`` even if the name stays the same); or
* they use the form ``from library import *`` which means all names
  from that library are exported.

And if I'm reading that correctly, you would only use from X import Y as Y if Y should be a public import, but I'm not really sure what this accomplishes.

[hauntsaninja]

The psutil stub also contains from typing import Any. Should type checkers accept the code psutil.Any? The only way to know is if there are some conventions for denoting what imports are incidental to the implementation vs actually part of a module's public interface. The from X import Y as Y is one such convention.

(Note that it looks to me like the psutil stubs were largely auto-generated. Stub auto-generators can struggle to figure out the public vs private distinction, so often default to re-exporting all imports, e.g. what is likely happening here: https://github.com/python/typeshed/blob/c0030b2b12738aefbbe71b857c20776b09d34e64/stubs/psutil/psutil/_psosx.pyi#L3)

[HunterAP23]

Ah so if I'm understanding this correctly, one of the imports in psutil/psutil/__init__.pyi mentioned above is from ._common import CONN_SYN_SENT as CONN_SYN_SENT.

If it was just from ._common import LINUX then psutil.CONN_SYN_SENT would be considered importable in other libraries - which it's not supposed to be.

So to let the type checking know whether or not it's meant to be a public or private import, the import has to be from X import Y as Y instead of from X import Y