What are typing.IO, TextIO, and BinaryIO good for?

[JelleZijlstra]

When should I use typing.IO, typing.TextIO, and typing.BinaryIO? Should I use Protocols instead? What is the difference between IO[str] and TextIO?

I'm starting this topic to get the ball rolling on this new forum and because I think it will give some illuminating insights about the type system. I'll write up some of my own thoughts below and I'd be happy to see insights from other people involved in typing.

[srittau]

My take: typing.IO, typing.TextIO, and typing.BinaryIO should be avoided if possible. For return types, the actual, concrete type that is returned should be used. For example:

import io

class MyFileWrapper(io.StringIO):
    ...

def my_opener() -> MyFileWrapper:  # not typing.IO[str] or typing.TextIO
    return MyFileWrapper()

Until the use of protocols for I/O arguments is widespread, these concrete classes often still need to derive from IO, for compatibility.

When returning an object returned by open(), typeshed's open() annotations are a good cheat sheet. Accordingly for object returned by other library functions.

For argument types, protocols should be used. These should be fairly tight, according to the actual needs of the function. They can either use predefined protocols, custom protocols, or an intersection of those:

import typing
if typing.TYPE_CHECKING:
    from _typeshed import SupportsRead

class NamedReader(SupportsRead[str]):
    @property
    def filename(self) -> str: ...

def read_and_print(stream: NamedReader) -> None:
    print(f"Reading {stream.filename}")
    print(stream.read())

Addendum (after reading Jelle's post): When defining protocols or base classes, it often makes sense to use a protocol for the return type. This documents what callers can expect the actual implementation to return, but doesn't constrain sub-classes to use a specific implementation.

Historically, typing's I/O types have proven to be quite problematic. The difference between IO vs TextIO and BinaryIO often was not clear (the latter are more specialized sub-classes), they are missing some necessary fields and methods, but have much too many required methods when all you want to do is call .read(). The concept of file-like is unfortunately not well-defined in Python, but still much used. This meant slight incompatibilities between all these "file-like" objects that could not be expressed using just IO etc., defeating the purpose of the type system when it comes to IO classes. Using protocols fixes this.

[JelleZijlstra]

These classes were meant to model the concept of a "file object", but when you start looking at how "file objects" are really used across the Python library system, you quickly realize that there's a ton of variation in what exactly you can do with them. There's certainly a read method and a write method, but does write return an int or None, and is the second argument to read positional-only or not? And what about all the other methods you might want, such as readline or seek?

Text and binary IO

typing.IO is generic over AnyStr because it is common in Python to have IO that is either bytes-based, returning raw bytes, or str-based, returning text decoded using some specific encoding (usually UTF-8 these days). For the former, you would use IO[bytes], and for the latter, IO[str].

But we also have BinaryIO and TextIO. Those are almost exactly the same as IO[bytes] and IO[str], as you can see at https://github.com/python/typeshed/blob/master/stdlib/typing.pyi#L483, but TextIO adds some properties that only make sense for text-based IO (such as encoding), and both override __enter__ (but I think that will make little difference in practice).

None of these classes are currently Protocols in typing. I think this is because they are so big that it is hardly practical to fully implement them without inheritance. typing.Sequence similarly isn't a Protocol.

typeshed conventions

In typeshed, where we provide type stubs for the standard library and for many third-party packages, our policy is now to avoid the typing.IO classes.

For example, if an API calls for a "file object" in an argument but actually just calls read on it, we would use a Protocol that just defines a read method. We find that most use cases just need a few common methods, so we put some common reusable Protocols in the _typeshed utility module. The advantage is that we won't generate spurious errors when someone passes a type that is accepted at runtime, but that doesn't fully implement typing.IO.

For return types, we try to use concrete types that match exactly what gets returned at runtime. For example, we created a complicated set of overloads for builtins.open that enables type checkers to pick the exact return type that will be used at runtime. This makes sure type checkers get to work with the most precise types and avoids incorrect errors when users perform some operation that works only with the concrete types.

I would recommend also using typeshed's conventions if you are providing types or stubs for a library. However, it may make sense to also use Protocol return types if you want to have more freedom in how to change your return types in the future.

Application conventions

If you are writing application code, typeshed's conventions can also make sense, but I think it is more defensible to use IO directly. Creating a custom Protocol is a lot of overhead, and if you control the whole application it is easy to adjust the types later if you want to pass some type that doesn't quite fit the IO mold.

[srittau]

[...] There's certainly a read method and a write method, [...]

This outlines another problem with the I/O classes: They are all both read and writer classes, but clients are usually only interested in either reading or writing. Providing a dummy write() method when you implement a "memory reader" (for example), is just unnecessary overhead.

[gvanrossum]

I agree that protocols are the way to go. I think this was an area of PEP 484 where we knew something was needed but we didn't quite know what to do and we just modeled what the runtime had. A big blocker was the lack of protocols at the time.

There's one niggling issue with using the minimal protocol: it constrains how a library may evolve. For example, suppose version 1.0 of a library has a function that reads a file using f.readline(), so we specify a protocol that just has readline(). But now version 2.0 adds an optimization where it uses f.read() to read the entire file and then uses .spitlines() on that. Or perhaps it uses f.__iter__() instead of f.readline(). Or maybe it uses f.seek() in addition to f.readline().

IOW, specifying one of the IO classes constrains the callers but allows the library to evolve, while specifying a minimal protocol gives callers maximal freedom while constraining the future evolution of the library. It's not always clear from reading the docs whether the library intends to evolve or not. Of course, duck typing means that users write code that works with the current version rather than code that is future-proof anyway, so my concern is mostly theoretical (hence my agreement in the first paragraph).

[srittau]

There's one niggling issue with using the minimal protocol: it constrains how a library may evolve.

I think that's one symptom of a larger issue: Should type annotations and stubs represent (documented) API or implementation? At the moment, at least in typeshed, we are leaning heavily on the "implementation" side. This is also the reason why we include undocumented or "private" (underscored) members on request. Personally, I wouldn't mind going a bit further in the "documented API" direction, and for non-typeshed projects my recommendation is to include methods you might reasonable start to use during the current major version of the library in the protocol.