[python-package] cut unnecessary files out of sdist package

[jameslamb]

Hello from Chicago 👋

Recently in LightGBM, I've been working on reducing the size of our Python package's source distribution. I found that many extra files from git submodules were being bundled in the package. This can be problematic in storage sensitive environments. For example, the first time I tried to use lightgbm + pandas + scikit-learn together on AWS Lambda, I had to do some surgery to trim out unnecessary things, to avoid hitting the 250 MB limit for extra packages (see description of microsoft/LightGBM#3579 if you're curious).

Cutting the package size could also help PyPi's data transfer costs a little bit 😀

I cut the size of lightgbm's sdist package by making the rules in MANIFEST.in more specific, to target only the files that were needed. You can see the diffs for the PRs below:

• [python] remove unnecessary files to reduce sdist size microsoft/LightGBM#3579
• [python-package] remove unnecessary files to reduce sdist size microsoft/LightGBM#3639
• [python-package] remove unused Eigen files, compile with EIGEN_MPL2_ONLY (fixes #3684) microsoft/LightGBM#3685

I can see that there are some files in xgboost that are not necessary. For example, all of the dmlc-core unit test code and even dmlc-core's .git/ directory are currently bundled in the package produced by python setup.py sdist.

how I'm checking the contents of the package (click me)

# with a clone of the repo
git submodule update --recursive
cd python-package
python setup.py sdist
open xgboost.egg-info/SOURCES.txt

# or from PyPi
wget https://files.pythonhosted.org/packages/8e/cd/c1c48514cdd03d735d38d2de471474eb7adc53fc5278cb4a877a25a29976/xgboost-1.3.1.tar.gz -O xgboost.tar.gz

tar -xvf xgboost.tar.gz
open xgboost-1.3.1/xgboost.egg-info/SOURCES.txt

I'd be happy to do this same work for the xgboost Python package, making the MANIFEST.in rules more specific to trim out unnecessary files. Would you consider a PR that did something similar?

Thanks for your time and consideration.

[hcho3]

@jameslamb Yes, it would be nice to remove unnecessary files from the source distribution. I will make sure to review your pull request.