Enable Installation of Python Package with System lib in a Virtual Environment

[oliverholworthy]

Goal

Enable Installation of Python Package with System lib in a Virtual Environment

Details

• Replaces sys.prefix with sys.base_prefix
  • these are different when inside a virtual environment. Using sys.base_prefix instead of sys.prefix enables installation of package in a virtual environment when the xgboost is already installed on the system

sys.base_prefix
Set during Python startup, before site.py is run, to the same value as prefix. If not running in a virtual environment, the values will stay the same; if site.py finds that a virtual environment is in use, the values of prefix and exec_prefix will be changed to point to the virtual environment, whereas base_prefix and base_exec_prefix will remain pointing to the base Python installation (the one which the virtual environment was created from).

Motivation

Setting up tests to run inside a virtual environment in an image that has xgboost pre-installed as a system lib. (e.g. nvcr.io/nvidia/tensorflow:23.06-tf2-py3). When in a virtual environment, the sys.prefix changes to point to the lib directory inside the virtualenvironment instead of the system lib path.

• Installing xgboost with the use_system_libxgboost=True in a virtual environment
• Using a tox virtual environment with sitepackages=true to pickup an xgboost installation present on the system already

[trivialfis]

Thank you for the work on improved support with venv. However, I got a bit confused by the paths now. Maybe we need a new option that's more general as discussed in #6706, specifically #6706 (comment). Without a new option, it's difficult to handle the diversity of packaging methods. In the case of venv, the shared object can be in /usr/lib or myvenv/lib, or myvenv/lib/python3.x/site-packages/xgboost/lib.

[hcho3]

There are times when we need to use sys.prefix. Example: libxgboost is installed inside a Conda enviornment. This PR will break the conda-forge recipe for XGBoost.

[trivialfis]

I think mamba always copies the python executable, so the base_prefix and the prefix are the same:

Python 3.10.12 | packaged by conda-forge | (main, Jun 23 2023, 22:40:32) [GCC 12.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.base_prefix
'/home/jiamingy/.anaconda/envs/xgboost_dev'
>>> sys.prefix
'/home/jiamingy/.anaconda/envs/xgboost_dev'

I expect conda to have the same behavior.

[hcho3]

It would be nice to make paths user configurable

[hcho3]

I realize that this PR doesn't break Conda. For now this change is fine. In a future PR, we can add user configuration for paths.

[trivialfis]

I canceled the job for the stalling test. It's caused by an issue in column-split test, which has been fixed in master branch. Will rebase the PR.

[trivialfis]

@hcho3 @oliverholworthy After reading the mamba document: https://mamba.readthedocs.io/en/latest/advanced_usage/more_concepts.html#linking I think this PR might be problematic if there are users who prefer soft-links for working across file system partitions.

[hcho3]

Thanks, I opened #9376 as a reminder.