[SPARK-49181] Remove site/docs/{version}/api/python/_sources folders and util/build-error-docs.py

[yaooqinn]

This PR removes interim data under the _sources folder for each version listed below:

./site/docs/4.0.0-preview1/api/python/_sources
./site/docs/3.1.2/api/python/_sources
./site/docs/3.3.1/api/python/_sources
./site/docs/3.3.0/api/python/_sources
./site/docs/3.1.3/api/python/_sources
./site/docs/3.4.0/api/python/_sources
./site/docs/3.2.2/api/python/_sources
./site/docs/3.4.1/api/python/_sources
./site/docs/3.2.4/api/python/_sources
./site/docs/3.2.3/api/python/_sources
./site/docs/3.5.0/api/python/_sources
./site/docs/3.1.1/api/python/_sources
./site/docs/3.3.2/api/python/_sources
./site/docs/3.5.1/api/python/_sources
./site/docs/3.3.3/api/python/_sources
./site/docs/3.3.4/api/python/_sources
./site/docs/3.2.1/api/python/_sources
./site/docs/3.4.3/api/python/_sources
./site/docs/3.2.0/api/python/_sources
./site/docs/3.4.2/api/python/_sources

After removing them, dangling links like:
https://spark.apache.org/docs/3.5.1/api/python/_sources/user_guide/sql/index.rst.txt are invisible from end users.

We also remove util/build-error-docs.py, which is a tool for doc-gen not for users from in 4.0.0-preview1.

The main goal of this PR is to remove unnecessary publications from doc to reduce the repo size of spark-website

[yaooqinn]

cc @HyukjinKwon @cloud-fan @dongjoon-hyun @srowen thanks

[srowen]

The idea here is simply that these are unused and so deleting them helps a little bit with the space crunch?
That's OK if so.

How did you find them, out of curiosity?

[cloud-fan]

shall we update Spark scripts to clean these files after doc building?

[yaooqinn]

The idea here is simply that these are unused and so deleting them helps a little bit with the space crunch?
That's OK if so.
How did you find them, out of curiosity?

Hi @srowen, I have gained some experience using reStructuredText and Sphinx in other Apache projects, e.g. Apache Kyuubi.

shall we update Spark scripts to clean these files after doc building?

Hi @cloud-fan apache/spark#47686 is also ready

[cloud-fan]

how much space we are saving here?

[yaooqinn]

about 10-20M for each version

[dongjoon-hyun]

+1, LGTM.

[dongjoon-hyun]

Yes, I also double-checked.

$ du -h 3.5.1/api/python/_sources
 68K	3.5.1/api/python/_sources/development
148K	3.5.1/api/python/_sources/user_guide/pandas_on_spark
 40K	3.5.1/api/python/_sources/user_guide/sql
208K	3.5.1/api/python/_sources/user_guide
 20K	3.5.1/api/python/_sources/migration_guide
2.9M	3.5.1/api/python/_sources/reference/pyspark.sql/api
3.0M	3.5.1/api/python/_sources/reference/pyspark.sql
180K	3.5.1/api/python/_sources/reference/pyspark.ss/api
196K	3.5.1/api/python/_sources/reference/pyspark.ss
2.5M	3.5.1/api/python/_sources/reference/api
3.0M	3.5.1/api/python/_sources/reference/pyspark.pandas/api
3.1M	3.5.1/api/python/_sources/reference/pyspark.pandas
8.8M	3.5.1/api/python/_sources/reference
4.1M	3.5.1/api/python/_sources/getting_started
 13M	3.5.1/api/python/_sources

[dongjoon-hyun]

Oh, @yaooqinn . It seems that this PR deleted the following.

$ COLUMNS=1000 git diff HEAD~1 --stat | grep -v '/api/python/_sources'
 site/docs/4.0.0-preview1/util/build-error-docs.py                                                                                              |   152 ---------
 33288 files changed, 1194166 deletions(-)

This PR:
https://github.com/apache/spark-website/blob/14b7c9e0fb1dd3945672b1fde7fedd4f9b1e585d/site/docs/4.0.0-preview1/util/build-error-docs.py

asf-site:
https://github.com/apache/spark-website/blob/4fd6a38/site/docs/4.0.0-preview1/util/build-error-docs.py

[dongjoon-hyun]

I manually do the same thing as a test and share it with you as a new PR. The result is not the same with this PR, @yaooqinn . In other words, this PR deletes a wrong file like util/build-error-docs.py which is outside of _sources and also deletes less number of files.

• Remove api/python/_sources/ #545

$ find . -name _sources
./site/docs/4.0.0-preview1/api/python/_sources
./site/docs/3.1.2/api/python/_sources
./site/docs/3.3.1/api/python/_sources
./site/docs/3.3.0/api/python/_sources
./site/docs/3.1.3/api/python/_sources
./site/docs/3.4.0/api/python/_sources
./site/docs/3.2.2/api/python/_sources
./site/docs/3.4.1/api/python/_sources
./site/docs/3.2.4/api/python/_sources
./site/docs/3.2.3/api/python/_sources
./site/docs/3.5.0/api/python/_sources
./site/docs/3.1.1/api/python/_sources
./site/docs/3.3.2/api/python/_sources
./site/docs/3.5.1/api/python/_sources
./site/docs/3.3.3/api/python/_sources
./site/docs/3.3.4/api/python/_sources
./site/docs/3.2.1/api/python/_sources
./site/docs/3.4.3/api/python/_sources
./site/docs/3.2.0/api/python/_sources
./site/docs/3.4.2/api/python/_sources

$ find . -name _sources | xargs rm -rf

[yaooqinn]

Thank you @dongjoon-hyun

I update the PR title to mention util/build-error-docs.py too. Spark main has moved it into _plugins already

[yaooqinn]

FYI, apache/spark@81948bb

[dongjoon-hyun]

Thank you for the fix.

[dongjoon-hyun]

I update the PR title to mention util/build-error-docs.py too. Spark main has moved it into _plugins already

BTW, for the above part, you are unable to delete that file by using apache/spark@81948bb because it's already published Apache Spark 4.0.0-preview1 artifacts.

https://github.com/apache/spark-website/blob/4fd6a38/site/docs/4.0.0-preview1/util/build-error-docs.py

I'm wondering how did you generate this PR initially. Could you revise the PR description by describing the reproducible steps, @yaooqinn ?

[yaooqinn]

BTW, for the above part, you are unable to delete that file by using apache/spark@81948bb because it's already published Apache Spark 4.0.0-preview1 artifacts.

I think it's similar to the "_sources" folders. Both of them are accessible via doc releases, but useless to users.

[dongjoon-hyun]

Sure, it's similar, of course, @yaooqinn . So, I gave +1 already.

BTW, please revise the PR description about your process to decide them.

I'm wondering how did you generate this PR initially. Could you revise the PR description by describing the reproducible steps, @yaooqinn ?

[yaooqinn]

Thank you @dongjoon-hyun, updated