Fix decoding multi part extension

[lhoestq]

e.g. a field named url.txt should be a treated as text

I also included a small fix to support .npz correctly

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[lhoestq]

takign the liberty to merge this for the viewer and a new dataset being released

[github-actions]

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

Benchmark: benchmark_array_xd.json

metric	read_batch_formatted_as_numpy after write_array2d	read_batch_formatted_as_numpy after write_flattened_sequence	read_batch_formatted_as_numpy after write_nested_sequence	read_batch_unformated after write_array2d	read_batch_unformated after write_flattened_sequence	read_batch_unformated after write_nested_sequence	read_col_formatted_as_numpy after write_array2d	read_col_formatted_as_numpy after write_flattened_sequence	read_col_formatted_as_numpy after write_nested_sequence	read_col_unformated after write_array2d	read_col_unformated after write_flattened_sequence	read_col_unformated after write_nested_sequence	read_formatted_as_numpy after write_array2d	read_formatted_as_numpy after write_flattened_sequence	read_formatted_as_numpy after write_nested_sequence	read_unformated after write_array2d	read_unformated after write_flattened_sequence	read_unformated after write_nested_sequence	write_array2d	write_flattened_sequence	write_nested_sequence
new / old (diff)	0.005004 / 0.011353 (-0.006349)	0.003352 / 0.011008 (-0.007657)	0.063035 / 0.038508 (0.024527)	0.032031 / 0.023109 (0.008922)	0.244801 / 0.275898 (-0.031097)	0.270622 / 0.323480 (-0.052857)	0.003110 / 0.007986 (-0.004876)	0.002629 / 0.004328 (-0.001700)	0.048784 / 0.004250 (0.044534)	0.045779 / 0.037052 (0.008726)	0.258642 / 0.258489 (0.000153)	0.291606 / 0.293841 (-0.002235)	0.028237 / 0.128546 (-0.100310)	0.010184 / 0.075646 (-0.065463)	0.202455 / 0.419271 (-0.216816)	0.036012 / 0.043533 (-0.007521)	0.248209 / 0.255139 (-0.006930)	0.267315 / 0.283200 (-0.015884)	0.019249 / 0.141683 (-0.122434)	1.120420 / 1.452155 (-0.331735)	1.169515 / 1.492716 (-0.323201)

Benchmark: benchmark_getitem_100B.json

metric	get_batch_of_1024_random_rows	get_batch_of_1024_rows	get_first_row	get_last_row
new / old (diff)	0.095193 / 0.018006 (0.077187)	0.300544 / 0.000490 (0.300055)	0.000214 / 0.000200 (0.000014)	0.000050 / 0.000054 (-0.000004)

Benchmark: benchmark_indices_mapping.json

metric	select	shard	shuffle	sort	train_test_split
new / old (diff)	0.019001 / 0.037411 (-0.018411)	0.061857 / 0.014526 (0.047331)	0.073379 / 0.176557 (-0.103178)	0.121293 / 0.737135 (-0.615843)	0.075665 / 0.296338 (-0.220673)

Benchmark: benchmark_iterating.json

metric	read 5000	read 50000	read_batch 50000 10	read_batch 50000 100	read_batch 50000 1000	read_formatted numpy 5000	read_formatted pandas 5000	read_formatted tensorflow 5000	read_formatted torch 5000	read_formatted_batch numpy 5000 10	read_formatted_batch numpy 5000 1000	shuffled read 5000	shuffled read 50000	shuffled read_batch 50000 10	shuffled read_batch 50000 100	shuffled read_batch 50000 1000	shuffled read_formatted numpy 5000	shuffled read_formatted_batch numpy 5000 10	shuffled read_formatted_batch numpy 5000 1000
new / old (diff)	0.285153 / 0.215209 (0.069944)	2.875527 / 2.077655 (0.797873)	1.479851 / 1.504120 (-0.024269)	1.360691 / 1.541195 (-0.180504)	1.385581 / 1.468490 (-0.082909)	0.566312 / 4.584777 (-4.018465)	2.400202 / 3.745712 (-1.345510)	2.719241 / 5.269862 (-2.550620)	1.706469 / 4.565676 (-2.859208)	0.062129 / 0.424275 (-0.362146)	0.005291 / 0.007607 (-0.002316)	0.334585 / 0.226044 (0.108540)	3.293347 / 2.268929 (1.024419)	1.790490 / 55.444624 (-53.654134)	1.505519 / 6.876477 (-5.370958)	1.527730 / 2.142072 (-0.614343)	0.644554 / 4.805227 (-4.160673)	0.119775 / 6.500664 (-6.380889)	0.056912 / 0.075469 (-0.018557)

Benchmark: benchmark_map_filter.json

metric	filter	map fast-tokenizer batched	map identity	map identity batched	map no-op batched	map no-op batched numpy	map no-op batched pandas	map no-op batched pytorch	map no-op batched tensorflow
new / old (diff)	0.977512 / 1.841788 (-0.864275)	11.293883 / 8.074308 (3.219575)	9.669439 / 10.191392 (-0.521953)	0.129910 / 0.680424 (-0.550514)	0.014322 / 0.534201 (-0.519879)	0.284967 / 0.579283 (-0.294316)	0.265355 / 0.434364 (-0.169008)	0.321965 / 0.540337 (-0.218372)	0.415254 / 1.386936 (-0.971682)

PyArrow==latest

Show updated benchmarks!

Benchmark: benchmark_array_xd.json

metric	read_batch_formatted_as_numpy after write_array2d	read_batch_formatted_as_numpy after write_flattened_sequence	read_batch_formatted_as_numpy after write_nested_sequence	read_batch_unformated after write_array2d	read_batch_unformated after write_flattened_sequence	read_batch_unformated after write_nested_sequence	read_col_formatted_as_numpy after write_array2d	read_col_formatted_as_numpy after write_flattened_sequence	read_col_formatted_as_numpy after write_nested_sequence	read_col_unformated after write_array2d	read_col_unformated after write_flattened_sequence	read_col_unformated after write_nested_sequence	read_formatted_as_numpy after write_array2d	read_formatted_as_numpy after write_flattened_sequence	read_formatted_as_numpy after write_nested_sequence	read_unformated after write_array2d	read_unformated after write_flattened_sequence	read_unformated after write_nested_sequence	write_array2d	write_flattened_sequence	write_nested_sequence
new / old (diff)	0.005138 / 0.011353 (-0.006215)	0.003321 / 0.011008 (-0.007687)	0.049731 / 0.038508 (0.011223)	0.032307 / 0.023109 (0.009198)	0.266331 / 0.275898 (-0.009567)	0.290863 / 0.323480 (-0.032617)	0.004151 / 0.007986 (-0.003835)	0.002684 / 0.004328 (-0.001644)	0.048760 / 0.004250 (0.044510)	0.042251 / 0.037052 (0.005199)	0.280414 / 0.258489 (0.021925)	0.305089 / 0.293841 (0.011248)	0.029118 / 0.128546 (-0.099428)	0.010276 / 0.075646 (-0.065370)	0.057790 / 0.419271 (-0.361482)	0.033290 / 0.043533 (-0.010243)	0.267250 / 0.255139 (0.012111)	0.285233 / 0.283200 (0.002034)	0.018587 / 0.141683 (-0.123096)	1.136198 / 1.452155 (-0.315957)	1.185274 / 1.492716 (-0.307442)

Benchmark: benchmark_getitem_100B.json

metric	get_batch_of_1024_random_rows	get_batch_of_1024_rows	get_first_row	get_last_row
new / old (diff)	0.096355 / 0.018006 (0.078349)	0.301827 / 0.000490 (0.301337)	0.000216 / 0.000200 (0.000016)	0.000052 / 0.000054 (-0.000003)

Benchmark: benchmark_indices_mapping.json

metric	select	shard	shuffle	sort	train_test_split
new / old (diff)	0.022607 / 0.037411 (-0.014805)	0.075724 / 0.014526 (0.061198)	0.088197 / 0.176557 (-0.088359)	0.127864 / 0.737135 (-0.609271)	0.089294 / 0.296338 (-0.207044)

Benchmark: benchmark_iterating.json

metric	read 5000	read 50000	read_batch 50000 10	read_batch 50000 100	read_batch 50000 1000	read_formatted numpy 5000	read_formatted pandas 5000	read_formatted tensorflow 5000	read_formatted torch 5000	read_formatted_batch numpy 5000 10	read_formatted_batch numpy 5000 1000	shuffled read 5000	shuffled read 50000	shuffled read_batch 50000 10	shuffled read_batch 50000 100	shuffled read_batch 50000 1000	shuffled read_formatted numpy 5000	shuffled read_formatted_batch numpy 5000 10	shuffled read_formatted_batch numpy 5000 1000
new / old (diff)	0.289321 / 0.215209 (0.074112)	2.832456 / 2.077655 (0.754802)	1.559208 / 1.504120 (0.055088)	1.426229 / 1.541195 (-0.114966)	1.424564 / 1.468490 (-0.043926)	0.557754 / 4.584777 (-4.027023)	0.940179 / 3.745712 (-2.805533)	2.713640 / 5.269862 (-2.556222)	1.697583 / 4.565676 (-2.868093)	0.062024 / 0.424275 (-0.362251)	0.005270 / 0.007607 (-0.002337)	0.339450 / 0.226044 (0.113406)	3.333024 / 2.268929 (1.064096)	1.946087 / 55.444624 (-53.498537)	1.601057 / 6.876477 (-5.275420)	1.599862 / 2.142072 (-0.542210)	0.642838 / 4.805227 (-4.162390)	0.120470 / 6.500664 (-6.380194)	0.040815 / 0.075469 (-0.034654)

Benchmark: benchmark_map_filter.json

metric	filter	map fast-tokenizer batched	map identity	map identity batched	map no-op batched	map no-op batched numpy	map no-op batched pandas	map no-op batched pytorch	map no-op batched tensorflow
new / old (diff)	1.012904 / 1.841788 (-0.828884)	11.917035 / 8.074308 (3.842727)	9.717822 / 10.191392 (-0.473570)	0.141730 / 0.680424 (-0.538694)	0.015750 / 0.534201 (-0.518451)	0.284470 / 0.579283 (-0.294813)	0.125662 / 0.434364 (-0.308702)	0.380740 / 0.540337 (-0.159598)	0.418119 / 1.386936 (-0.968817)