pdm.lock files produced by Renovate are specific to the CPU architecture on which RenovateBot runs

[sh-at-cs]

What would you like help with?

I think I found a bug

How are you running Renovate?

Self-hosted

If you're self-hosting Renovate, tell us which platform (GitHub, GitLab, etc) and which version of Renovate.

GitLab, Renovate 38.21.1

Please tell us more about your question or problem

We're running Renovate for a Python project using the PDM package manager on machines with aarch64 CPUs, but our application itself is meant to run on amd64 CPUs.

Unfortunately, it seems like the pdm.lock files created by Renovate specify the CPU architecture:

...

[[metadata.targets]]
requires_python = "==3.11.9"
platform = "manylinux_2_31_aarch64"
implementation = "cpython"
gil_disabled = false

This then causes issues in subsequent CI pipelines, e.g. those of Renovate's own Lock file maintenance MRs, which, like our application itself, run on amd64 CPUs. Specifically, pdm install fails with:

1.406 STATUS: Resolving packages from lockfile...
1.456 ERROR: None of the lock targets matches the current env (==3.11.9, manylinux_2_36_x86_64, cpython):
1.456  - (==3.11.9, manylinux_2_31_aarch64, cpython)
1.458 [PdmException]: No compatible lock target found

As far as I can tell, PDM comes with functionality to specify a target platform (including architecture) when generating the lockfile.

Does Renovate expose a configuration option or something like that which would allow us to use this functionality to override the mismatching host architecture? And if not, would it be possible to implement this in Renovate's PDM support?

Thanks in advance!

[rarkins]

Renovate does not have this functionality. Sounds like it could "learn" it from the existing lock file without needing the user to manually configure though. Can you create a reproduction repo and a "suggest an idea" discussion? Perhaps having your reproduction as aarch64 would be best, then we can test it with amd64 in dev.

[sh-at-cs]

As you suggested, I've created a "reverse" reproduction (project pipeline runs on Mac OS/aarch64, Renovate on Linux/amd64) here: renovate-repro-pdm-arch-mismatch
You can see an error equivalent to the one I quoted with only architectures reversed in the CI logs of the Lock file maintenance PR created by Renovate.

While preparing this, I noticed two things:

1. This whole feature of PDM's seems pretty new. The relevant docs page was only committed last month. In the codebase where we first noticed this issue, our previous lockfile didn't contain the [[metadata.targets]] block at all, which I've now simulated in the reproduction repo by just removing it manually.
2. This issue isn't specific to CPU architecture mismatches, but actually any slight mismatch in the Python platform version will cause the same error. E.g. if I have manylinux_2_36_x86_64 on my local machine and on my CI machines I run pdm install but they happen to have manylinux_2_35_x86_64, it'll also report that error and fail.

Given these findings, I'm currently more inclined to say this should be solved on PDM's side, because I can't imagine that this is the way this feature is meant to work. I'll link the PDM issue here when/if I end up creating one.

[sh-at-cs]

All right, riddle solved. This issue has nothing to do with Renovate whatsoever, so Renovate devs can stop reading here.

---

For anyone else who encounters this and finds this thread via a search engine, here is the full story:

PDM only writes platform-specific [[metadata.targets]] sections when both of these conditions are met:

1. There is no [[metadata.targets]] section yet (i.e. the lock file must have been created with a version of PDM < 2.17) and
2. the strategy field of the [metadata] section of pdm.lock doesn't contain "cross-platform".

If strategy contains "cross-platform", newer versions of PDM will add a platform-agnostic [[metadata.targets]] section and remove "cross-platform" from strategy. And if you start from even older versions of PDM that didn't have the strategy field, then "intermediate" versions will automatically add strategy = ["cross-platform"] for you as well.

So by creating a pdm.lock file with a newer version of PDM and then deleting the [[metadata.targets]] section for the reproduction, I was "cheating" because that shouldn't be a situation one ought to get into over the course of using different PDM versions.

But then how did the codebase where we originally encountered this issue get into this state? Well, at one point we updated our lockfile, which by that point was recent enough to have a [[metadata.targets]] section and strategy = [] (no longer ["cross-platform"]), with an older version of PDM. Apparently, when you update lockfiles with older PDM versions, they just remove all the things they don't understand. So it promptly removed [[metadata.targets]] but left strategy = [] as-is. And that's what produced just the situation above. Moral of the story is to only touch metadata with monotonically increasing versions of package managers or prepare for trouble.

PDM issue I created in the hope that they can make it more robust with respect to such things in the future: pdm-project/pdm#3121