Project Links in left navbar display in reverse order from setup.py project_urls

[medmunds]

The Project Links displayed in the left navbar appear to be in the opposite order as they're listed in setup project_urls (and resulting package METADATA).

This isn't a huge issue, but some project urls are usually better starting points than others, so it would be nice to keep them ordered.

Example:

setup(
    # ...
    project_urls=OrderedDict([
        ("1-Tracker", "https://github.com/anymail/django-anymail/issues"),
        ("2-Changelog", "https://github.com/anymail/django-anymail/releases"),
        ("3-Source", "https://github.com/anymail/django-anymail"),
        ("4-Documentation", "https://anymail.readthedocs.io/en/%s/" % release_tag),
    ]),
    # ...
)

Result on test warehouse:

I've checked that twine is posting the project_urls fields in the same order as they're listed in setup.py, so I'm pretty sure this is occurring in Warehouse. Here's a full upload from twine captured in Runscope, if helpful (not real auth, of course).

twine version 1.9.1, pkginfo: 1.4.1, requests: 2.18.4, setuptools: 38.5.1, requests-toolbelt: 0.8.0, tqdm: 4.19.6, Python: 3.6.4

[di]

Thanks for the report! Especially the runscope capture, that's really nice.

Since this is "working as expected" (i.e. the links are there), and since the spec doesn't say anything about ordering, I'm going to put the "Feature Request" label on this, which means that it's prioritized sometime after the completion of Milestone 6 (unless someone else wants to work on it in the meantime).

[medmunds]

Makes sense.

The Runscope capture will likely expire in 30 days or so; if it's gone, it was pretty easy to make:

twine upload --repository-url https://${YOUR_BUCKET}.runscope.net/pypi -u TEST -p TEST dist/* 

(with a free Runscope account, or substitute your preferred postbin service)

[dstufft]

I am pretty sure we just treat this as a dictionary so it might even change orders between page loads? Unless we added a sort in there somewhere.

[medmunds]

FWIW, there does seem to be an effort in models.Release.urls to ensure some ordering of urls is maintained—though it looks like this is more about Homepage at the top and Download at the bottom. And its test has a TODO to check the ordering. [Edit: sorry, models.Release.urls, not models.Project.urls]

[di]

We're treating them as Dependencys, so the order that we're getting is from newest to oldest, that's why they're reversed.

@medmunds Sounds like you're halfway towards solving this yourself. 🙂 The next step would be fixing that TODO and making the test fail (and also getting tests running locally), want to take a shot at it? Feel free to ask questions here if you have any.

[medmunds]

@di sure, I'll take a shot. (But probably won't get around to getting a dev environment set up before next week.)

I don't see where Dependency or Release.project_urls specifies an order, and I don't see an obvious Dependency column to sort on anyway, so I'm guessing "newest to oldest" just happens to be what the DB is returning. (But I'm a SQLAlchemy novice. Am I missing something? Implicit primary key that could be used for order_by?)

An easy, but brittle, fix is just reversing the project_urls returned from Release.urls. (Brittle as in DB-dependent, though no less deterministic than the current ordering. 🙂)

A robust fix is probably adding a position column as part of "TODO: Move project URLs into their own table, since they are not actually a 'dependency'." (Which is probably more than I'm ready to bite off just yet.)

[di]

(But probably won't get around to getting a dev environment set up before next week.)

No worries! Like I said, this isn't a huge priority.

so I'm guessing "newest to oldest" just happens to be what the DB is returning.

Indeed, I'm pretty sure this is what's happening. Here is where they're created.

A robust fix is probably adding a position column as part of "TODO: Move project URLs into their own table, since they are not actually a 'dependency'." (Which is probably more than I'm ready to bite off just yet.)

Yeah, which means that this probably needs to wait until post-legacy shutdown when we are more free to change the way things work in the DB.

[medmunds]

OK, opened a PR for the simple fix, and updated/added comments for the more-robust fix later when there are related changes going into the DB.

(Btw, everything—code and tests—had already been using OrderedDicts, except for the one TODO line in the test case which deliberately suppressed the ordering because of this issue.)

[cas--]

Please revisit fixing this issue, the PR looked good to me...

[medmunds]

(The earlier PR was a fragile workaround, and was withdrawn.)

So the correct fix involves changing the DB schema to move Project URLs out of Release Dependencies, and adding a position column to support ordering. (See this TODO in models.py: "Move project URLs into their own table, since they are not actually a 'dependency'.")

For the schema change: would it make sense to create a generalized ReleaseMetadata table (key/value/position) that could support other types of (non-dependency, possibly ordered) per-release metadata? Or just keep it simple with a ProjectURL table (url/position) for now. (I think Project-URL is the only metadata this would currently apply to.) @di?

Incidentally, the tests for project URL ordering already exist, but right now they deliberately ignore order by casting OrderedDict to dict. (See this TODO in test_models.py.)

[dstufft]

I would just make it a dedicated table for project URLs. Alternatively you could do something like using a jsonb column or something, but you'd have to put your own validations in then to make sure the structure is sane.

[nedbat]

Any chance of fixing this? I discovered it myself and adapted, but especially now that dicts are ordered, it's baffling that it's reverse order. I noticed it again on https://pypi.org/project/asgiref which has urls of Changelog / Further Documentation / Documentation... :)

[di]

We should probably just alphabetize this list for now instead, as there's nothing in the metadata specs, build tools, etc that supports explicitly ordered project URLs or guarantees ordering.

but especially now that dicts are ordered

This might be true for the Python version you're using for your build, but it's not guaranteed to be true for all build environments for all artifacts on PyPI, and not the only thing that influences the order here. IMO, trying to continue to support this is fragile and likely just going to require more maintenance in the future.

[pelson]

Looks to me that latest warehouse is sorting the URLs (except homepage) - I think this issue may therefore be resolved?

[nedbat]

They do seem sorted after Homepage now. It would be great to have control over the order, but the metadata formats don't specify them in an ordered way.

[miketheman]

Indeed, the order has been made explicit when this PR added a new table for them.
#11564