removed scipy from build-only dependencies #3538

filip-komarzyniec · 2024-06-14T13:36:41Z

Resolves: #3536

In addition to removing scipy from build-system.requires table I've also updated the README file.

Please refer to the comments for some additional doubts I have regarding more changes that might need to be introduced.

filip-komarzyniec · 2024-06-14T13:42:53Z

README.md

+This software depends on [NumPy], a Python package for
+scientific computing. You must have it installed prior to installing


Does it really have to be installed prior to installing gensim having in mind that a proper numpy version is already specified in build-system.requires table? Meaning it will be downloaded automatically during gensim build phase.

If it does not --- what should we do with regard to the next paragraph about BLAS lib? I think it will not be customisable for any architecture that a numpy .whl build exists for.

Yes, I think the sentence starting "You must" is incorrect. Numpy will get installed automatically if it already isn't there.

With BLAS, I think we just leave that note as-is. You can't customize it as part of the gensim install. If you want to mess with BLAS, then you have to do it yourself, before installing numpy and gensim.

Right @piskvorky ?

I forgot how exactly it worked back then. IIRC numpy was needed for the installation of gensim: as in numpy actively used during install, down to its C API. Maybe scipy too. I believe that is the intent behind "You must have [numpy] installed prior to installing gensim".

If pip installs numpy automatically before the gensim install itself starts, that's great. Some people don't use pip, I don't know all the deployment methods people use (from source, conda, etc) and whether they all install dependencies first.

In any case that was 15 years ago, the current ecosystem might work differently, maybe none of this is needed.

As I see it, the problem of linking Numpy to any BLAS lib is a matter to worry about only during building Numpy from source.
This can occur only if:

one wants to customise his Numpy build in a non typical way. In this case I believe such person knows what he's doing so this note is not really valuable for him,

one installs gensim which in turn downloads appropriate NumPy build (but src distribution instead of a .whl). Then the pip (or I believe any package manager) would try building the NumPy from source and implicitly linking an existing BLAS lib.

What's more, a NumPy installed as a gensim's dependency will not be (in some cases) the same NumPy as used for building gensim. Pip by default builds a library in a separate temporary virtual env where it installs build-only dependencies.

I've redacted the paragraph. Please review whether it sounds sensible .

… machines

mpenkov · 2024-06-16T21:46:44Z

pyproject.toml

+    # oldest supported Numpy for non-'arm64|aarch64' architectures is 1.17.*
+    # as well as for 'Windows' on arm64. The oldest supported Numpy by Gensim
+    # is 1.18.5. Remove below line when they increase the oldest supported Numpy for those platforms
+    "numpy==1.18.5; python_version=='3.8' and platform_machine != 'aarch64' and (platform_machine != 'arm64' or platform_system == 'Windows')",


What is the motivation behind changing this?

Looking at the meta package oldest-supported-numpy I see that numpy<1.18.5 will be downloaded for python 3.8 and Windows on arm64 (apart from all non-arm64 and non-aarch64 platforms).

So I believe this check was incomplete resulting in installing unsupported version of Numpy for python3.8 arm64 Windows users.

@piskvorky Do we want to deal with this now? It hasn't been reported yet, so it may not actually be a problem. It's been a while since I've messed with oldest-supported-numpy, so I'm not 100% sure this fix is needed and safe.

I understand the reasoning.
Should I revert this change then?

Yes, let's revert it. Feel free to make a separate PR with that change so we can review it later, after the current release is out.

I'll do as advised, thanks!

… Windows machines" This reverts commit 68197fb.

mpenkov

Good work, thank you!

filip-komarzyniec · 2024-06-28T10:03:33Z

I see the builds failing on tests step (due to different scipy versions being installed for a given python version).

My PR should not affect anything more than library building step (which passes successfully each time).
Is there anything that can be done in order to fix the tests (for example should I merge my changes with any specific other branch fixing scipy runtime dependency?)

mpenkov · 2024-06-28T14:20:00Z

Very strange. I see your previous commits built fine. I will investigate.

juhoinkinen · 2024-07-16T10:46:56Z

@mpenkov and @filip-komarzyniec, I tried restricting scipy versions to <1.14 in setup.py, and this makes the CICD tests pass, see this run.

~~Not sure why scipy 1.13.1 got installed for Python 3.8 and 3.9 in the first place (this logs line), and scipy 1.14.0 for Python 3.10+ (this logs line). I checked that:~~

~~pipdeptree showed that there is a second-degree-dependency of scipy via visdom, but it has no version constraints,~~
~~in PyPI there is just the source distribution of visdom, so no Python version effects from there either.~~

Edit: Well, ofcourse the difference for Python versions comes because scipy 1.14.0 supports only Python 3.10+, as stated in the release notes.

Anyway, I wonder could this scipy version constraint <1.14 be applied or is there something more to it?

(I took a look because our CICD pipeline is not working due to Gensim installation issues.)

mpenkov · 2024-07-18T02:49:50Z

I think I see what's happening here:

We don't need scipy to build wheels
We do need scipy to test the wheels; more specifically 1.13.1 or below, because future versions got rid of sparsetools

…ove_scipy_from_build_only_deps' into filip-komarzyniec_3536_remove_scipy_from_build_only_deps

juhoinkinen · 2024-07-18T06:07:18Z

Previously I got the tests to pass by pinning to numpy<2.0 for Python 3.12 build-system, see this run: https://github.com/juhoinkinen/gensim/actions/runs/9953367669

(Ref #3541 (comment)).

mpenkov · 2024-07-18T11:12:10Z

The problem with scipy is fixed, now we have to solve the problem with macos.

https://github.blog/changelog/2024-05-20-actions-upcoming-changes-to-github-hosted-macos-runners/

Crosswind · 2024-07-18T11:32:19Z

Let's see if macos-12 will fix the issue. It may not.
I have found some information on the problem which seems to originate in pip. The build uses a pretty outdated version of it.
There is pypa/cibuildwheel#1410 and then this PR bloomberg/blazingmq-sdk-python#23 which could be worth a try.

mpenkov · 2024-07-18T12:02:51Z

Alright, looks like we're green. Thanks everyone!

removed scipy from build-only dependencies; updated README accordingly

a148e9b

filip-komarzyniec commented Jun 14, 2024

View reviewed changes

filip-komarzyniec added 2 commits June 15, 2024 14:46

fixed conditions regarding oldest numpy installation on arm64 Windows…

68197fb

… machines

README rephrased regarding NumPy installation

1dc75de

filip-komarzyniec force-pushed the 3536_remove_scipy_from_build_only_deps branch from b170395 to 1dc75de Compare June 15, 2024 13:19

mpenkov reviewed Jun 16, 2024

View reviewed changes

Revert "fixed conditions regarding oldest numpy installation on arm64…

c3a1f4a

… Windows machines" This reverts commit 68197fb.

filip-komarzyniec mentioned this pull request Jun 27, 2024

gensim on Windows arm64 system probably downloads unsupported NumPy version during building from source #3539

Open

mpenkov approved these changes Jun 27, 2024

View reviewed changes

mpenkov added 5 commits July 18, 2024 11:52

pin scipy, need old sparsetools for wheel testing

1d4beef

Merge branch 'develop' into 3536_remove_scipy_from_build_only_deps

843d05a

make scipy pin more strict

ee1e3dc

Merge remote-tracking branch 'refs/remotes/filip-komarzyniec/3536_rem…

9256f85

…ove_scipy_from_build_only_deps' into filip-komarzyniec_3536_remove_scipy_from_build_only_deps

use macos-latest instead of deprecated macos-11

8df0205

mpenkov added 3 commits July 18, 2024 15:28

pin scipy in setup.py

420a415

remove redundant scipy pin from build-wheels.yml

5db44ee

get rid of remaining macos-11 traces in CI workflow

0b01e9e

mpenkov mentioned this pull request Jul 18, 2024

gensim package import issues #3544

Closed

This was referenced Jul 18, 2024

Pin scipy in installation requirements as well to fix failing tests #3547

Closed

Update Windows and Ubuntu runners to -latest #3549

Open

use macos-12 instead of macos-latest

3b1f09e

mpenkov merged commit 79f982b into piskvorky:develop Jul 18, 2024
32 checks passed

juhoinkinen mentioned this pull request Jul 18, 2024

Installation with Poetry fails on Python 3.12 - error: ‘PyArray_Descr’ {aka ‘struct _PyArray_Descr’} has no member named ‘subarray’ #3541

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

removed scipy from build-only dependencies #3538

removed scipy from build-only dependencies #3538

filip-komarzyniec commented Jun 14, 2024

filip-komarzyniec Jun 14, 2024

mpenkov Jun 15, 2024

piskvorky Jun 15, 2024 •

edited

Loading

filip-komarzyniec Jun 15, 2024

mpenkov Jun 16, 2024

filip-komarzyniec Jun 17, 2024

mpenkov Jun 17, 2024

filip-komarzyniec Jun 22, 2024

mpenkov Jun 23, 2024

filip-komarzyniec Jun 24, 2024

mpenkov left a comment

filip-komarzyniec commented Jun 28, 2024

mpenkov commented Jun 28, 2024

juhoinkinen commented Jul 16, 2024 •

edited

Loading

mpenkov commented Jul 18, 2024

juhoinkinen commented Jul 18, 2024 •

edited

Loading

mpenkov commented Jul 18, 2024

Crosswind commented Jul 18, 2024 •

edited

Loading

mpenkov commented Jul 18, 2024

		This software depends on [NumPy], a Python package for
		scientific computing. You must have it installed prior to installing

removed scipy from build-only dependencies #3538

removed scipy from build-only dependencies #3538

Conversation

filip-komarzyniec commented Jun 14, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

piskvorky Jun 15, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mpenkov left a comment

Choose a reason for hiding this comment

filip-komarzyniec commented Jun 28, 2024

mpenkov commented Jun 28, 2024

juhoinkinen commented Jul 16, 2024 • edited Loading

mpenkov commented Jul 18, 2024

juhoinkinen commented Jul 18, 2024 • edited Loading

mpenkov commented Jul 18, 2024

Crosswind commented Jul 18, 2024 • edited Loading

mpenkov commented Jul 18, 2024

piskvorky Jun 15, 2024 •

edited

Loading

juhoinkinen commented Jul 16, 2024 •

edited

Loading

juhoinkinen commented Jul 18, 2024 •

edited

Loading

Crosswind commented Jul 18, 2024 •

edited

Loading