multipart dependency conflicts with python-multipart, making impossible to install treq and starlette on the same system

[mgorny]

There are currently two multipart implementations on PyPI: multipart and python-multipart. Both install the same multipart package, and therefore cannot be installed simultaneously.

Both starlette and synapse use (apparently more popular) python-multipart. This is probably understandable, given that prior to June, the last release of multipart was in 2021. However, treq now started using multipart, which means it is no longer possible to install treq and starlette on the same system.

Please consider using python-multipart for the sake of compatibility instead.

[glyph]

Perhaps python-multipart should change its import name, since it postdates multipart? The apache license isn't a dealbreaker, but it's nice that multipart is MIT, has more recent commits, has a push parser in progress, and has currently-passing CI.

Neither appears to have any documentation, which isn't great, but multipart's README is more comprehensive and doesn't contain broken links

[mgorny]

python-multipart changing at this point would break all its reverse dependencies.

Honestly, this isn't my fight. All I'm pointing out that this change is making synapse not correctly installable, since it simultaneously depends on python-multipart and treq (which now depends on multipart).

Furthermore, since multipart is just a .py file, while python-multipart is a full package, if both are installed (pip doesn't do anything to prevent that) the latter will take precedence.

Yes, Python ecosystem is completely broken by design.

[glyph]

python-multipart changing at this point would break all its reverse dependencies.

Honestly, this isn't my fight. All I'm pointing out that this change is making synapse not correctly installable, since it simultaneously depends on python-multipart and treq (which now depends on multipart).

Yeah, thanks for letting us know.

Furthermore, since multipart is just a .py file, while python-multipart is a full package, if both are installed (pip doesn't do anything to prevent that) the latter will take precedence.

OK, good to know.

Yes, Python ecosystem is completely broken by design.

"completely" is a strong word there, but this is certainly a flaw.

[glyph]

Given that things are in a broken state currently, I am going to do the following:

• vendor multipart since (as you note) it is a single file, so we don't need to change the code for an immediate fix
• file the bug upstream with multipart, here: Name conflict with https://github.com/Kludex/python-multipart defnull/multipart#53
• file the bug upstream with python-multipart, here: package fight! rename to avoid import conflict with https://github.com/defnull/multipart/ Kludex/python-multipart#152

[glyph]

@mgorny OK, vendoring branch done. Perhaps if @twm has some time we can get a review & release soon.

[defnull]

I answered the request for renaming multipart.py in the issue linked above. tl;dr: no

Vendoring seems to be a sensible workaround for now, given how easy it is with a single-file no-dependency module. I'd suggest watching the github project for new releases though, as we have a pretty nice new sansio (non-blockling) parser almost ready that is significantly faster than the current implementation (or werkzeug) and also more predictable in memory and CPU consumption. Documentation will also improve.

[glyph]

@defnull For what it's worth, I'm inclined to agree with your reasoning, but we are in this situation due to decisions of others that I have relatively limited influence over :). We'll see what the python-multipart maintainers have to say. Best-case scenario here would be to merge the projects, since it's not clear to me that there's a great advantage of one over the other?

[defnull]

Merging both projects is also not an option, as the APIs are completely different. The only solution I can think of is for python-multipart to give up their conflicting package name and release a new version (with a major version bump, because it's a breaking change), then deprecate the old release and package. But no one can force anyone to do anything in this ecosystem.

Anyway, multipart is in a way better shape now. I'm actually happy enough with the current state that I'm planning a 1.0 release. That might change the 'popularity' aspect in the future a bit. Maybe. I'm really bad at promoting my own projects, so we'll see.

[Kludex]

This issue looks like a competition to see which project is doing better... "Which one has better documentation?" "Which one has a better parse?"

If it's possible to do what python-multipart did, then the problem is with the ecosystem itself. 🤷‍♂️

Anyway, it looks like this was solved. @mgorny we've interacted on many issues already, and I think I've always tried to be helpful. If you need something else here, let me know. 👍

[twm]

Hi @Kludex, treq maintainer here.

Anyway, it looks like this was solved. @mgorny we've interacted on many issues already, and I think I've always tried to be helpful. If you need something else here, let me know. 👍

To be clear, this has only been solved temporarily, for treq in the PyPI ecosystem. Vendoring multipart is not a long-term solution (there has already been a release — our vendored copy is out of date!). Nor is it a complete solution: when Debian goes to package treq I expect that they will un-vendor multipart and we'll be right back to the same conflict.

Can you offer some timeline for when you plan to merge python-multipart into Starlette? I don't see an open issue for doing so but I'd like to know when I can un-vendor multipart.

[Kludex]

I can't offer timeline. Sorry.

[glyph]

@Kludex I understand that you can't give a timeline, but if we did the work to vendor python-multipart into startlette, or port starlette to multipart instead, would you accept that PR?

[Kludex]

We need to clear the list of PRs in python-multipart, and make it easy to just copy paste onto starlette i.e. increase test coverage to 100%, fully type check, and match the lint rules.

Moving to multipart doesn't really makes any sense... I don't understand that suggestion.

[eli-schwartz]

If it's possible to do what python-multipart did, then the problem is with the ecosystem itself. 🤷‍♂️

Not really sure how you concluded this. Do you think it is a problematic aspect of the python ecosystem, that:

• a single wheel can install multiple modules?
• packages are permitted to provide import names that differ from the PyPI name?
  • packages can be available as sdist, and install modules based on running python code, which PyPI cannot check regardless and can therefore be used to circumvent any technical measure designed to enforce the previous point?

Why would it be a good outcome for PyPI to desire to enforce any such thing? It's a community cooperation issue. There are plenty of good reasons to allow it as a possibility for people who believe they have good reasons to reuse the same import name. Common reasons why a project might choose to do so include:

• renaming a project when you own both the old and new PyPI names
• forking a project whose author died to provide continuity
• forking a project whose author dropped the project entirely, and invited the community to fork under a new name due to concerns about dependency trust which preclude giving someone they do not know, control over a trusted name
• offering an API-compatible drop-in replacement for a package with enhancements that haven't been accepted by the original project
• providing an alternative policy implementation for a policy-focused package such as certifi

As for python-multipart:

I don't think anyone here doubts that python-multipart knew, back when it was first published, that it was conflicting with the import name of an existing project. This acknowledgment is implicit in the fact that python-multipart was unable to name itself "multipart" and chose to upload with the prefix "python-" indicating "python-multipart is written in the python programming language" to distinguish itself from "multipart: a module uploaded to the index of python software".

Of course, python-multipart could have chosen to name its module as "python_multipart" (or perhaps pymultipart). But it did not do so, for whatever reason. I guess you're saying it was your right to do so in a manner that doesn't really seem to fit into any of the options I mentioned above, because PyPI didn't have a technological restriction preventing you from doing so? That is definitely one way of looking at it.

Anyway, it looks like this was solved. @mgorny we've interacted on many issues already, and I think I've always tried to be helpful. If you need something else here, let me know. 👍

I don't know the context here but from a naive perspective this looks oddly patronizing, especially given that as far as I can tell your attempt to be helpful consists of confirming that you do not intend to do anything to solve the problem up to and including providing a timeline for decommissioning the package and merging it into your other project -- which I'd typically understand to mean that you do not want other people to use python-multipart at all and it exists as an internal implementation detail of starlette.

So I really wonder why it isn't a sufficient solution, to rename the module to import _starlette_multipart and solve this headache for everyone, including the people for whom bundling an old and outdated version of multipart isn't a solution, but rather an additional (vetting and compliance) problem, such as the aforementioned Debian.

...

To be clear: I'm not saying you owe me something as an open source project and that I demand an action of you. I just find it super weird that you popped up on this issue purely to state for the record that you "always try to be helpful". It feels like some kind of PR stunt but I am having a hard time unwrapping the meaning contained within it.

[defnull]

We need to clear the list of PRs in python-multipart, and make it easy to just copy paste onto starlette i.e. increase test coverage to 100%, fully type check, and match the lint rules.

If python-multipart is really just for starlette and not intended to be used as a dependency by others, then please rename the package and add a clear warning to its README. You control both projects, changing the import name to starlette_multipart should be possible within a single patch release of starlette. There is no reason to wait another 6 years.

If python-multipart is intended to be used by not just starlette, then also, please rename it. Breaking changes are fine for a 0.0.x library, most dependant packages should have an exact version pinned and not break without warning. The fix for dependent packages would be trivial. Fixing the current name conflict situation, however, is almost impossible without vendoring one of the two.

[mgorny]

Just I wanted to say once again, a big "thank you" to you for pushing towards resolving the bigger issue. With python-multipart finally renaming itself, things are hopefully going to settle sometime soon.

[glyph]

With python-multipart finally renaming itself

Is this officially decided? Kludex/python-multipart#16 looks like it's still up in the air

[merwok]

Yes, a new module name was added: Kludex/python-multipart#166

(The backward compat causes issues with installing and reloading in same process without invalidating importlib caches, so there’s a second PR: Kludex/python-multipart#168)

[glyph]

I saw the second PR but didn't realize the context. Thank you, this is great!

[Kludex]

I had to yank the release that contained the first approach.

[glyph]

@Kludex I realize that the compatibility puzzle is tricky here, but as long as you've approved the idea of the change, I'm sure we will converge on a strategy that works :)