[python-package] Install lightgbm from source failed

[WeichenXu123]

Description

Installing lightgbm from github source failed:

pip install git+https://github.com/microsoft/LightGBM.git#subdirectory=python-package

Output:

+ pip install git+https://github.com/microsoft/LightGBM.git#subdirectory=python-package
Collecting git+https://github.com/microsoft/LightGBM.git#subdirectory=python-package
  Cloning https://github.com/microsoft/LightGBM.git to /tmp/pip-req-build-6yedz97p
  Running command git clone --filter=blob:none --quiet https://github.com/microsoft/LightGBM.git /tmp/pip-req-build-6yedz97p
  Resolved https://github.com/microsoft/LightGBM.git to commit 90786966ae7e75850043b38a12b1775176463435
  Running command git submodule update --init --recursive -q
  Preparing metadata (setup.py): started
  Preparing metadata (setup.py): finished with status 'error'
  error: subprocess-exited-with-error
  
  × python setup.py egg_info did not run successfully.
  │ exit code: 1
  ╰─> [12 lines of output]
      Traceback (most recent call last):
        File "<string>", line 2, in <module>
        File "<pip-setuptools-caller>", line 34, in <module>
        File "/tmp/pip-req-build-6yedz97p/python-package/setup.py", line 300, in <module>
          version = _version_file.read_text(encoding='utf-8').strip()
        File "/opt/hostedtoolcache/Python/3.8.12/x64/lib/python3.8/pathlib.py", line 1[23](https://github.com/mlflow/mlflow/actions/runs/4907369819/jobs/8762445726#step:11:24)6, in read_text
          with self.open(mode='r', encoding=encoding, errors=errors) as f:
        File "/opt/hostedtoolcache/Python/3.8.12/x64/lib/python3.8/pathlib.py", line 1222, in open
          return io.open(self, mode, buffering, encoding, errors, newline,
        File "/opt/hostedtoolcache/Python/3.8.12/x64/lib/python3.8/pathlib.py", line 1078, in _opener
          return self._accessor.open(self, flags, mode)
      FileNotFoundError: [Errno 2] No such file or directory: '/tmp/pip-req-build-6yedz97p/python-package/lightgbm/VERSION.txt'
      [end of output]

Reproducible example

Environment info

LightGBM version or commit hash:

Command(s) you used to install LightGBM

pip install git+https://github.com/microsoft/LightGBM.git#subdirectory=python-package

python_version=3.8
OS: Ubuntu 22.04.2 LTS

Additional Comments

[jameslamb]

Thanks for using LightGBM.

Installing the development version of lightgbm that way is not supported.

To build an unreleased version, please run the following.

git clone --recursive https://github.com/microsoft/LightGBM
cd LightGBM
sh build-python.sh install

[WeichenXu123]

Thank you. We previously use pip install git+https://github.com/microsoft/LightGBM.git#subdirectory=python-package in our mlflow CI. If it is no longer supported, we can switch to the approach you recommend. :)

[jameslamb]

Ah sorry about that!

To give you some more context...

This was broken by me in #5837. Since setuptools and the other Python packaging tools are discontinuing support for invoking python setup.py install, we are working on cutting over to a PEP 517/518 compliant backend.

Those backends rely on packaging in static metadata + tend to assume that you are building from the root of a repo that is only a Python package. That isn't true here in LightGBM, where we have a tree layout like this:

├── CMakeLists.txt
├── R-package/
├── README.md
├── VERSION.txt
├── cmake/
├── external_libs/
├── include/
├── python-package/
           ├── lightgbm/
                       ├──  __init__.py
                       ├── *.py
           ├── pyproject.toml
           ├── setup.cfg
├── src/
├── swig/
├── tests/
└── windows/

You can read much more about this topic at #5061.

Please feel free to tag me on any PRs over in mlflow related to LightGBM if want some help there. And of course we'd welcome code contributions and input on design decisions like this from anyone over in mlflow who's maintaining LightGBM compatibility.

[WeichenXu123]

@jameslamb Thank you!
So I can still use pip install git+https://github.com/microsoft/LightGBM.git#subdirectory=python-package to install dev version once you fix the issue ?

[jameslamb]

No, we are not planning to support pip install git+ going forward, just as we have not supported remotes::install_github() in the R package for several years.

To be honest, I didn't consider direction installation with pip install git+ when working on #5837. It's not something we recommend in any of the project's documentation.

We do really appreciate you installing the latest development version of lightgbm in mlflow's CI, since that can be a valuable source of bug reports and early feedback on issues prior to releases. But at this time our small team of maintainers here is focused on getting out a v4.0.0 of LightGBM (#5153) that is at least compliant with the impending changes in setuptools + pip as soon as possible.

I expected that pip install lightgbm and conda install lightgbm are far more common than using pip install git+.

[jameslamb]

Closing this for now, please @ any time if you need more help.

[mirekphd]

But @jameslamb it fails at python setup.py, which I believe is no longer supported, so isn't this the underlying reason (rather than git+ directly)?

[jameslamb]

fails at python setupy py... isn't this the underlying reason?

Not exactly. This issue was opened at a transition time where there was still a setupy.py file in this repo, but where it no longer could be directly invoked with python setupy.py install because logic in it which reached up into other directories was removed.

Nothing about using pip install git+ with sufficiently new versions of pip is opinionated about whether a project uses asetupy.py file or a pyproject.toml file to configure wheel-building.

The more important thing to understand here is this... to build lightgbm (the LightGBM Python package) from the sources here on GitHub, you now have to use the build-python.sh script at the root of the repo.

The changes to pip, setuptools, and build described in detail in #5061 drove us to make significant changes to how the Python package is built, and using a shell script seemed the easiest and most future-proof way to accomplish that. Many Python build tools assume that all files are at or below wherever the setupy.py / pyproject.toml is... which isn't true here in this project, since this repo simultaneously supports a Python package, R package, standalone CLI, standalone C++ shared library, and more.

[github-actions]

This issue has been automatically locked since there has not been any recent activity since it was closed. To start a new related discussion, open a new issue at https://github.com/microsoft/LightGBM/issues including a reference to this.