Do not use Parquet exports if revision is passed

[albertvillanova]

Fix #6554.

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[albertvillanova]

I have also removed passing revision to fs.open: 3acefe5

• Previous condition makes this unnecessary
• Additionally, note that this has never worked (the revision is ignored). I have opened a PR to fix it:
  • Pass revision in path to AbstractBufferedFile init huggingface_hub#1948

[lhoestq]

good catch !

[lhoestq]

As shared on slack, HubDatasetModuleFactoryWithParquetExport raises a DatasetsServerError already if the user tries to load another revision that the one from the parquet export. And therefore it fall backs on using HubDatasetModuleFactoryWithScript

[albertvillanova]

@lhoestq I would say that although current implementation finally returns HubDatasetModuleFactoryWithScript as expected, with this PR we avoid the useless call to HubDatasetModuleFactoryWithParquetExport.get_module, so this is more optimal.

[lhoestq]

Indeed, lgtm then :)

[github-actions]

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

Benchmark: benchmark_array_xd.json

metric	read_batch_formatted_as_numpy after write_array2d	read_batch_formatted_as_numpy after write_flattened_sequence	read_batch_formatted_as_numpy after write_nested_sequence	read_batch_unformated after write_array2d	read_batch_unformated after write_flattened_sequence	read_batch_unformated after write_nested_sequence	read_col_formatted_as_numpy after write_array2d	read_col_formatted_as_numpy after write_flattened_sequence	read_col_formatted_as_numpy after write_nested_sequence	read_col_unformated after write_array2d	read_col_unformated after write_flattened_sequence	read_col_unformated after write_nested_sequence	read_formatted_as_numpy after write_array2d	read_formatted_as_numpy after write_flattened_sequence	read_formatted_as_numpy after write_nested_sequence	read_unformated after write_array2d	read_unformated after write_flattened_sequence	read_unformated after write_nested_sequence	write_array2d	write_flattened_sequence	write_nested_sequence
new / old (diff)	0.005596 / 0.011353 (-0.005757)	0.004022 / 0.011008 (-0.006986)	0.064041 / 0.038508 (0.025533)	0.030683 / 0.023109 (0.007574)	0.245236 / 0.275898 (-0.030662)	0.269657 / 0.323480 (-0.053823)	0.003142 / 0.007986 (-0.004844)	0.002821 / 0.004328 (-0.001507)	0.048774 / 0.004250 (0.044523)	0.043771 / 0.037052 (0.006719)	0.258202 / 0.258489 (-0.000287)	0.288381 / 0.293841 (-0.005460)	0.028154 / 0.128546 (-0.100392)	0.011071 / 0.075646 (-0.064576)	0.209836 / 0.419271 (-0.209436)	0.035923 / 0.043533 (-0.007609)	0.248361 / 0.255139 (-0.006777)	0.268728 / 0.283200 (-0.014472)	0.019982 / 0.141683 (-0.121701)	1.172330 / 1.452155 (-0.279824)	1.192262 / 1.492716 (-0.300455)

Benchmark: benchmark_getitem_100B.json

metric	get_batch_of_1024_random_rows	get_batch_of_1024_rows	get_first_row	get_last_row
new / old (diff)	0.089231 / 0.018006 (0.071225)	0.299192 / 0.000490 (0.298702)	0.000214 / 0.000200 (0.000014)	0.000054 / 0.000054 (-0.000001)

Benchmark: benchmark_indices_mapping.json

metric	select	shard	shuffle	sort	train_test_split
new / old (diff)	0.018358 / 0.037411 (-0.019053)	0.062633 / 0.014526 (0.048107)	0.076276 / 0.176557 (-0.100280)	0.120862 / 0.737135 (-0.616274)	0.075958 / 0.296338 (-0.220380)

Benchmark: benchmark_iterating.json

metric	read 5000	read 50000	read_batch 50000 10	read_batch 50000 100	read_batch 50000 1000	read_formatted numpy 5000	read_formatted pandas 5000	read_formatted tensorflow 5000	read_formatted torch 5000	read_formatted_batch numpy 5000 10	read_formatted_batch numpy 5000 1000	shuffled read 5000	shuffled read 50000	shuffled read_batch 50000 10	shuffled read_batch 50000 100	shuffled read_batch 50000 1000	shuffled read_formatted numpy 5000	shuffled read_formatted_batch numpy 5000 10	shuffled read_formatted_batch numpy 5000 1000
new / old (diff)	0.291575 / 0.215209 (0.076366)	2.855908 / 2.077655 (0.778253)	1.459891 / 1.504120 (-0.044229)	1.374945 / 1.541195 (-0.166250)	1.333759 / 1.468490 (-0.134731)	0.575428 / 4.584777 (-4.009348)	2.414253 / 3.745712 (-1.331459)	2.768222 / 5.269862 (-2.501639)	1.705005 / 4.565676 (-2.860672)	0.063406 / 0.424275 (-0.360869)	0.004981 / 0.007607 (-0.002626)	0.343826 / 0.226044 (0.117781)	3.418143 / 2.268929 (1.149215)	1.856571 / 55.444624 (-53.588053)	1.571318 / 6.876477 (-5.305159)	1.609897 / 2.142072 (-0.532175)	0.646779 / 4.805227 (-4.158448)	0.118143 / 6.500664 (-6.382521)	0.042408 / 0.075469 (-0.033061)

Benchmark: benchmark_map_filter.json

metric	filter	map fast-tokenizer batched	map identity	map identity batched	map no-op batched	map no-op batched numpy	map no-op batched pandas	map no-op batched pytorch	map no-op batched tensorflow
new / old (diff)	0.965091 / 1.841788 (-0.876697)	11.569655 / 8.074308 (3.495347)	10.587818 / 10.191392 (0.396426)	0.128518 / 0.680424 (-0.551905)	0.013954 / 0.534201 (-0.520247)	0.287244 / 0.579283 (-0.292039)	0.263755 / 0.434364 (-0.170609)	0.321661 / 0.540337 (-0.218676)	0.428753 / 1.386936 (-0.958183)

PyArrow==latest

Show updated benchmarks!

Benchmark: benchmark_array_xd.json

metric	read_batch_formatted_as_numpy after write_array2d	read_batch_formatted_as_numpy after write_flattened_sequence	read_batch_formatted_as_numpy after write_nested_sequence	read_batch_unformated after write_array2d	read_batch_unformated after write_flattened_sequence	read_batch_unformated after write_nested_sequence	read_col_formatted_as_numpy after write_array2d	read_col_formatted_as_numpy after write_flattened_sequence	read_col_formatted_as_numpy after write_nested_sequence	read_col_unformated after write_array2d	read_col_unformated after write_flattened_sequence	read_col_unformated after write_nested_sequence	read_formatted_as_numpy after write_array2d	read_formatted_as_numpy after write_flattened_sequence	read_formatted_as_numpy after write_nested_sequence	read_unformated after write_array2d	read_unformated after write_flattened_sequence	read_unformated after write_nested_sequence	write_array2d	write_flattened_sequence	write_nested_sequence
new / old (diff)	0.005568 / 0.011353 (-0.005785)	0.003755 / 0.011008 (-0.007253)	0.049134 / 0.038508 (0.010626)	0.032113 / 0.023109 (0.009004)	0.276645 / 0.275898 (0.000747)	0.299240 / 0.323480 (-0.024240)	0.004297 / 0.007986 (-0.003689)	0.002727 / 0.004328 (-0.001602)	0.048420 / 0.004250 (0.044170)	0.045070 / 0.037052 (0.008017)	0.288597 / 0.258489 (0.030108)	0.320824 / 0.293841 (0.026983)	0.053293 / 0.128546 (-0.075253)	0.011002 / 0.075646 (-0.064644)	0.057747 / 0.419271 (-0.361524)	0.034389 / 0.043533 (-0.009143)	0.277914 / 0.255139 (0.022775)	0.292919 / 0.283200 (0.009719)	0.018252 / 0.141683 (-0.123431)	1.187245 / 1.452155 (-0.264910)	1.199823 / 1.492716 (-0.292893)

Benchmark: benchmark_getitem_100B.json

metric	get_batch_of_1024_random_rows	get_batch_of_1024_rows	get_first_row	get_last_row
new / old (diff)	0.088338 / 0.018006 (0.070332)	0.297498 / 0.000490 (0.297008)	0.000206 / 0.000200 (0.000006)	0.000048 / 0.000054 (-0.000007)

Benchmark: benchmark_indices_mapping.json

metric	select	shard	shuffle	sort	train_test_split
new / old (diff)	0.021445 / 0.037411 (-0.015966)	0.075522 / 0.014526 (0.060996)	0.086010 / 0.176557 (-0.090546)	0.124938 / 0.737135 (-0.612197)	0.087542 / 0.296338 (-0.208796)

Benchmark: benchmark_iterating.json

metric	read 5000	read 50000	read_batch 50000 10	read_batch 50000 100	read_batch 50000 1000	read_formatted numpy 5000	read_formatted pandas 5000	read_formatted tensorflow 5000	read_formatted torch 5000	read_formatted_batch numpy 5000 10	read_formatted_batch numpy 5000 1000	shuffled read 5000	shuffled read 50000	shuffled read_batch 50000 10	shuffled read_batch 50000 100	shuffled read_batch 50000 1000	shuffled read_formatted numpy 5000	shuffled read_formatted_batch numpy 5000 10	shuffled read_formatted_batch numpy 5000 1000
new / old (diff)	0.292460 / 0.215209 (0.077251)	2.841290 / 2.077655 (0.763635)	1.537941 / 1.504120 (0.033821)	1.409903 / 1.541195 (-0.131291)	1.435339 / 1.468490 (-0.033151)	0.578967 / 4.584777 (-4.005810)	2.398588 / 3.745712 (-1.347125)	2.662342 / 5.269862 (-2.607520)	1.743055 / 4.565676 (-2.822622)	0.064043 / 0.424275 (-0.360232)	0.005030 / 0.007607 (-0.002577)	0.348542 / 0.226044 (0.122498)	3.395854 / 2.268929 (1.126926)	1.918935 / 55.444624 (-53.525689)	1.639320 / 6.876477 (-5.237157)	1.740406 / 2.142072 (-0.401666)	0.653346 / 4.805227 (-4.151881)	0.117298 / 6.500664 (-6.383366)	0.040635 / 0.075469 (-0.034834)

Benchmark: benchmark_map_filter.json

metric	filter	map fast-tokenizer batched	map identity	map identity batched	map no-op batched	map no-op batched numpy	map no-op batched pandas	map no-op batched pytorch	map no-op batched tensorflow
new / old (diff)	1.008277 / 1.841788 (-0.833510)	12.069369 / 8.074308 (3.995061)	10.967322 / 10.191392 (0.775930)	0.131938 / 0.680424 (-0.548486)	0.015418 / 0.534201 (-0.518783)	0.297257 / 0.579283 (-0.282026)	0.270742 / 0.434364 (-0.163622)	0.332296 / 0.540337 (-0.208042)	0.421606 / 1.386936 (-0.965330)