NEXUS-43022: Group pypi repository for multiple proxy repositories is broken in 3.68.1

[yakirm-cr]

• What problem are you trying to solve?
  We are currently using Nexus OSS version 3.63 and we are trying to upgrade to a newer version (currently, 3.68.1).
  Unfortunately, Group pypi repository for multiple proxy repositories is broken in 3.68.1.

Here are the steps to reproduce the issue on Ubuntu 22.04 with docker:

1. Start Sonatype Nexus 3 via docker by running via terminal:
   docker run --rm -d --name nexus3 -p 8081:8081 sonatype/nexus3:3.68.1
2. Once Nexus is up (you can verify by running docker logs -f nexus3), login to it (you can obtain the admin password by running docker exec -ti nexus3 cat /nexus-data/admin.password) and follow the initial wizard.
3. Create the two proxy pypi repositories:
   i. pypi-proxy to https://pypi.python.org/.
   ii. piwheels-proxy to https://piwheels.org/.
4. Create a group pypi repository called pypi-all grouping the proxy repositories created in the previous step in the same order they are listed above - 1^st pypi-proxy and 2^nd piwheels-proxy.
5. Create a file called requirements.txt with the following context:

   attrs~=23.2.0
   bidict~=0.23.0
   cached-property~=1.5.2
   charset-normalizer~=3.3.0
   click~=8.1.3
   dnspython~=2.6.1
   fasteners~=0.18
   flasgger~=0.9.7.1
   flake8~=7.0.0
   Flask~=3.0.0
   Flask-Cors~=4.0.0
   Flask-SocketIO~=5.3.6
   gunicorn~=21.2.0
   importlib-metadata~=7.1.0
   importlib-resources~=6.4.0
   itsdangerous~=2.1.2
   Jinja2~=3.1.2
   jsonschema~=4.21.0
   MarkupSafe~=2.1.2
   mistune~=3.0.1
   netifaces~=0.11.0
   pip~=24.0
   pkgutil_resolve_name~=1.3.10
   pyrsistent~=0.20.0
   python-dateutil~=2.9.0
   python-engineio~=4.9.0
   python-socketio~=5.11.0
   PyYAML~=6.0
   referencing~=0.34.0
   requests~=2.31.0
   rpds-py~=0.18.0
   setuptools~=69.2.0
   six~=1.16.0
   urllib3~=2.2.0
   Werkzeug~=3.0.0
   zipp~=3.18.1

6. Run the following commands via terminal:

   python -m venv .env --prompt=test
   . .env/bin/activate
   pip install --no-cache --index-url=http://localhost:8081/repository/pypi-all/simple --trusted-host=localhost -r ./requirements.txt

Expected result:
All libraries listed in the requirments.txt are installed successfully in the Python virtual environment.

Actual results:
Installation of one of the libraries (random) fails with the following error.

Collecting fasteners~=0.18
  Downloading http://localhost:8081/repository/pypi-all/packages/fasteners/0.19/fasteners-0.19-py3-none-any.whl (18 kB)
ERROR: THESE PACKAGES DO NOT MATCH THE HASHES FROM THE REQUIREMENTS FILE. If you have updated the package versions, please update the hashes. Otherwise, examine the package contents carefully; someone may have tampered with them.
    fasteners~=0.18 from http://localhost:8081/repository/pypi-all/packages/fasteners/0.19/fasteners-0.19-py3-none-any.whl#sha256=bba5c854998fca7d8b6acde63344d6a241b0228973455f307fddf55bfda24743 (from -r ./requirements.txt (line 7)):
        Expected sha256 bba5c854998fca7d8b6acde63344d6a241b0228973455f307fddf55bfda24743
             Got        758819cb5d94cdedf4e836988b74de396ceacb8e2794d21f82d131fd9ee77237

• Do you have a workaround you are using at present?
  Remain in version 3.63.

• What feature or behavior is this required for?
  Grouping multiple pypi repositories together as a single point of access.

• How could we solve this issue? (Not knowing is okay!)
  Unfortunately, I do not know. 🙁

• Tell us about your Nexus Repository deployment: what version, operating system, and database are you using?
  We are running version 3.63 via Docker image sonatype/nexus3:3.63.0 (running on Ubuntu 22.04).
  In addition we are using the default internal OrientDB.

• Anything else?
  Nope.

[debug-richard]

I can confirm this issue with 3.68.0 and 3.68.1.
If a wheel is present in A and B the correct file from A is downloaded but the checksum from B is used.
It looks like the index got messed up, but rebuilding the indexes and invalidating the caches didn't help.
Since a downgrade from 3.68.0 is not possible, I would consider this an urgent issue.

[nblair]

Thanks @yakirm-cr and @debug-richard - we're looking at this and have setup an internal ticket to investigate.

[yakirm-cr]

@nblair
This issue is blocking us to upgrade to version 3.68.1 (and above).
Any update about it?
Please, advise.

[MrDaGree]

@nblair - any updates? I went to go upgrade my instance (non-docker) that serves quite a lot of internal users but am unable to given updates past 3.63 break the pypi simple page.

[debug-richard]

My workaround is to set up a second older/working instance to proxy the main instance for the pypi groups.
This doesn't really fix the problem, but if you can't downgrade, this is a temporary solution...

[vfazio]

Any updates on this? This is a rather critical feature that now no longer functions. It essentially prevents customers from taking any security fixes in more recent versions without sacrificing this functionality.

[MrDaGree]

Bump. @nblair - curious if any updates to this? Would really appreciate some kind of update even if its just "we havent got to it yet"

[vfazio]

I don't see this called out in the 3.69 or 3.70 release notes so i assume we're still waiting on a fix

[vfazio]

@mrprescott you're the only other sonatype rep I see responding to recent issues. Is there an update on this issue?

[mrprescott]

We're tracking this internally as NEXUS-43022. No updates or ETA as it hasn't made it to the top of the queue, but we will update here once it's resolved.