Deprecate and schedule removal of collections.abc.ByteString and typing.ByteString

[JelleZijlstra]

The current docstring of collections.abc.ByteString is:

    """This unifies bytes and bytearray.

    XXX Should add all their methods.
    """

Let's do that last thing. This will be useful for typing code that accepts both bytes and bytearray, especially with my proposal in PEP-688 to make bytes no longer acceptable as a shortcut for bytearray in the type system.

cc @rhettinger for collections.abc

Linked PRs

• gh-91896: Deprecate collections.abc.ByteString #102096
• gh-91896: Improve visibility of ByteString deprecation warnings #104294
• gh-91896: Fixup some docs issues following ByteString deprecation #104422
• gh-91896: Revert some very noisy DeprecationWarnings for ByteString #104424

[rhettinger]

The general rule is that we can never add methods to ABCs once they are published. The purpose of an ABC is promise that a minimal set of methods are available. If isinstance(x, SomeABC) returns true, the methods in the ABC are expected to be present. Adding methods presents a problem for existing code that has registered a class as being compliant with SomeABC. If it lacks the new methods, then the registered promise is invalid.

Perhaps because this is an empty ABC that is almost entirely unused, this might be okay. On the other hand, because it is an empty ABC that is almost entirely unused, there is almost nothing to be gained by adding it. AFAICT no one has ever asked for or needed this since the ABC was added 15 years ago. That suggests that there is no problem to be solved here.

[JelleZijlstra]

Perhaps because this is an empty ABC that is almost entirely unused, this might be okay.

Right, the ABC is hardly useful with no methods. I searched on grep.app and found no uses of ByteString.register except for one of memoryview in what appears to be an old copy of typing.py.

On the other hand, because it is an empty ABC that is almost entirely unused, there is almost nothing to be gained by adding it. AFAICT no one has ever asked for or needed this since the ABC was added 15 years ago. That suggests that there is no problem to be solved here.

Well, I'm asking for it now. The use case is type annotating code that accepts both bytes and bytearray.

[rhettinger]

If you were to do it correctly, the procedure would be to subclass ByteString and add the new methods in a subclass. Loosely, this is similar to why we had to add IterableUserDict in Python 2 rather than modifying the existing UserDict.

To annotate code that accepts both bytes or bytearray wouldn't the correct way be to write: b: bytes | bytearray just like you would with tuple | list?

One other thought: the name ByteString wasn't very good to begin as is suggests bytes | str. If you really must do this, it would be better to create a new, well-named ABC with the requisite methods and even leave the old ABC around (it isn't hurting anything) or deprecate it.

[rhettinger]

I forgot to mention that the docs promise something narrower that the bytes/bytearray API. It says, "ABCs for read-only and mutable sequences". There is no promise of the extra methods found in bytes or bytearray. Existing code reasonably by subclassing from ByteString and not providing or expecting any of the stringlike methods.

All around, I think this proposal is a contract violation and that a new ABC should be created. A scan on grep.app is insufficient to show this won't be a breaking change (most of the world's Python code isn't publicly visible).

[JelleZijlstra]

I forgot to mention that the docs promise something narrower that the bytes/bytearray API. It says, "ABCs for read-only and mutable sequences". There is no promise of the extra methods found in bytes or bytearray. Existing code reasonably by subclassing from ByteString and not providing or expecting any of the stringlike methods.

That documentation is for Sequence, MutableSequence, and ByteString together (https://docs.python.org/3/library/collections.abc.html#collections.abc.Sequence). "Read-only and mutable sequences" is a good description for the first two, but the documentation really doesn't tell me what ByteString is good for.

ByteString is also documented at https://docs.python.org/3/library/typing.html#typing.ByteString, but that documentation has a couple of problems:

• memoryview is not in fact registered as a ByteString
• ByteString is not in fact generic (and it doesn't make sense for it to be generic)

All around, I think this proposal is a contract violation and that a new ABC should be created. A scan on grep.app is insufficient to show this won't be a breaking change (most of the world's Python code isn't publicly visible).

That's a reasonable point. If we can't use the existing ByteString ABC for bytes | bytearray, I don't think it's worth creating a new ABC—as you said above, the union annotation is good enough.

But if we keep ByteString as is, with no methods, I have no idea what it's useful for. We occasionally get people trying to use the ABC in type annotations, so the current state causes confusion.

Perhaps we could deprecate ByteString, or explicitly document its limited use.

[rhettinger]

Perhaps we could deprecate ByteString, or explicitly document its limited use.

I vote for deprecation because the name is bad (implying bytes | str) and it would just be a continuing point of confusion.

[serhiy-storchaka]

The name is good to me. It implies the bytes-like object with str methods (find(), lower(), isspace(), etc).

The terms "buffer", "bytes-like", "bytestring" and "bytes string" are used loosely in the documentation, but there are several meanings of bytes-likeness:

• Supports the buffer protocol.
• Additionally supports len() which returns the size in bytes.
• Additionally supports indexing.
• Has most methods of str (except encode() of course).

Unfortunately there are no strongly defined terms and corresponding abstract classes, protocols or types in the code to express the requirements precisely.

[ankith26]

Hi! I'm one of the contributors to the pygame project.

I'm not sure whether this is the best best place to be asking, but it is relevant to the usage of ByteString.

So we have a function implemented with the python C API, and it uses y#, which according to the docs is a format string for a generic "bytes-like" sized object. The term "bytes-like" is defined here in the glossary which gives me the impression that any function using y# must accept a wide range of "byte-like" objects. Simply using bytes | bytearray would be narrow, and probably miss some kinds of objects. The same also mentions that "bytes-like object" is an object supporting the C level buffer protocol (which, is also not fully exposed on the python end in my understanding)

I was looking for a suitable ABC to typehint this, and the closest thing I could find that already exists is ByteString.
The next closest thing I found is typing.SupportsBytes which by the naming, gives me the impression that this is what I'm looking for, but weirdly enough, the concrete bytes object itself does not confirm to this protocol (due to missing the __bytes__ method)

I suppose a set of ABCs for the buffer protocol (if this were to be added) would be the closest replacement to ByteString, and would also work for my usecase (typing the C level y# arg format)

[TeamSpen210]

The problem with buffers is that it doesn't have a visible Python API, so the ABC would be really weird, being entirely empty. See previous discussion at python/typing#593 and then PEP 688.

[hauntsaninja]

This has been deprecated for 3.12