[bug] platform_machine reported wrong on Arm machines lead to Missing suitable source/wheel file entry errors

[heimalne]

Describe the issue

In our project we have a dependency that should use a normal package tensorflow on all platforms except on arm64 macs; there it should use a special, other package tensorflow-macos.

A raw poetry lock && poetry install without Nix works as expected on all machines.

However, on my arm64 M1 Mac poetry2nix tries to download the normal tensorflow package when it shouldn't:

$ nix-build ...
error: Missing suitable source/wheel file entry for tensorflow

May it have an issue with multiple markers?

Additional context

Running on an M1 arm64 Mac with Python 3.9.18 with a pyproject.toml:

...
[tool.poetry.dependencies]
tensorflow = {version = "2.9.0", markers = "sys_platform != \"darwin\" or platform_machine != \"arm64\""}
tensorflow-macos = {version = "2.9.0", markers = "sys_platform == \"darwin\" and platform_machine == \"arm64\""}
tensorflow-metal = {version = "0.4.0", markers = "sys_platform == \"darwin\" and platform_machine == \"arm64\""}

[heimalne]

There are actually two issues:

1. The default overrides for tensorflow-macos has an implicit dependency to the normal tensorflow package and expects it to be present

   poetry2nix/overrides/default.nix

   Line 3312 in 3c92540

   tensorflow-macos = super.tensorflow-macos.overridePythonAttrs (

   :

     tensorflow = super.tensorflow.overridePythonAttrs (
        _old: {
          postInstall = ''
            rm $out/bin/tensorboard
          '';
        }
      );

      tensorflow-macos = super.tensorflow-macos.overridePythonAttrs (
        _old: {
          inherit (self.tensorflow) postInstall; <------------------
        }
      );

2. The platform_machine reported by poetry2nix on an M1 Arm Mac is probably wrong and therefore the marker platform_machine == \"arm64\" evaluates differently than expected:

pep508Env = lib.debug.traceValSeq (pyproject-nix.lib.pep508.mkEnviron python);

from

poetry2nix/default.nix

Line 175 in 3c92540

pep508Env = pyproject-nix.lib.pep508.mkEnviron python;

results in

{
  implementation_name = { type = "string"; value = "cpython"; };
  implementation_version = { type = "version"; value = { dev = null; epoch = 0; local = null; post = null; pre = null; release = [ 3 9 18 ]; }; };
  os_name = { type = "string"; value = "posix"; };
  platform_machine = { type = "string"; value = "aarch64"; };    <----------------------------- should be arm64
  platform_python_implementation = { type = "string"; value = "CPython"; };
  platform_release = { type = "string"; value = ""; };
  platform_system = { type = "string"; value = "Darwin"; };
  platform_version = { type = "version"; value = { dev = null; epoch = 0; local = null; post = null; pre = null; release = [ ]; }; };
  python_full_version = { type = "version"; value = { dev = null; epoch = 0; local = null; post = null; pre = null; release = [ 3 9 18 ]; }; };
  python_version = {
    type = "version";
    value = { dev = null; epoch = 0; local = null; post = null; pre = null; release = [ 3 9 ]; };
  };
  sys_platform = { type = "string"; value = "darwin"; };
}

where as the wheel files metadata use arm64 tag, e.g. "tensorflow_macos-2.9.0-cp39-cp39-macosx_11_0_arm64.whl as well as:

$ python
Python 3.9.16 (main, Mar 27 2023, 10:43:20)
>>> import platform
>>> platform.machine()
'arm64'

[Pegasust]

Thanks for looking into this. I ran to this problem probably months ago when I then decided to give up when I discovered the nifty details for tensorflow-macos and decided to look into related PEPs (I feel asleep searching on manylinux). It's inevitable that our system-arch tuple is different from Python's, I forgot if it's advisable to create this such mapping.

This looks really detailed, is PR ready, or do you still need help investigating somewhere?

[heimalne]

No PR's ready. Feel free to copy and adapt the hacks at b8faf1f and 0efcdb9
I can't say when i will have time again to work on this.

Also some packages like tensorflow-macos and watchfiles change the way poetry2nix want's to build our pyproject.toml project; it tries instead to build them as a wheel (which it isn't) and fails like in #1510 No clue yet.

[heimalne]

As dependency markers are quite broken in the current state of poetry2nix on Arm Macs, maybe someone with better understanding of the codebase could take a look at creating a mapping from platform string "aarch64" to Python package string "arm64" at a good place in the code? 0efcdb9 is too hacky.
Maybe @cpcloud or @adisbladis?

[heimalne]

There are actually two issues:

1. The default overrides for tensorflow-macos has an implicit dependency to the normal tensorflow package and expects it to be present

@Pegasust fyi: This was fixed with #1689