Add filter desc

[lhoestq]

Otherwise it would show a Map progress bar, since it uses map under the hood

[HuggingFaceDocBuilderDev]

The documentation is not available anymore as the PR was closed or merged.

[github-actions]

Show benchmarks

PyArrow==6.0.0

Show updated benchmarks!

Benchmark: benchmark_array_xd.json

metric	read_batch_formatted_as_numpy after write_array2d	read_batch_formatted_as_numpy after write_flattened_sequence	read_batch_formatted_as_numpy after write_nested_sequence	read_batch_unformated after write_array2d	read_batch_unformated after write_flattened_sequence	read_batch_unformated after write_nested_sequence	read_col_formatted_as_numpy after write_array2d	read_col_formatted_as_numpy after write_flattened_sequence	read_col_formatted_as_numpy after write_nested_sequence	read_col_unformated after write_array2d	read_col_unformated after write_flattened_sequence	read_col_unformated after write_nested_sequence	read_formatted_as_numpy after write_array2d	read_formatted_as_numpy after write_flattened_sequence	read_formatted_as_numpy after write_nested_sequence	read_unformated after write_array2d	read_unformated after write_flattened_sequence	read_unformated after write_nested_sequence	write_array2d	write_flattened_sequence	write_nested_sequence
new / old (diff)	0.008477 / 0.011353 (-0.002875)	0.004565 / 0.011008 (-0.006443)	0.101640 / 0.038508 (0.063132)	0.029581 / 0.023109 (0.006472)	0.296524 / 0.275898 (0.020625)	0.363175 / 0.323480 (0.039695)	0.006961 / 0.007986 (-0.001024)	0.003365 / 0.004328 (-0.000963)	0.079689 / 0.004250 (0.075439)	0.034881 / 0.037052 (-0.002171)	0.310979 / 0.258489 (0.052489)	0.348663 / 0.293841 (0.054822)	0.034549 / 0.128546 (-0.093997)	0.011463 / 0.075646 (-0.064184)	0.326218 / 0.419271 (-0.093053)	0.041393 / 0.043533 (-0.002140)	0.297604 / 0.255139 (0.042465)	0.335751 / 0.283200 (0.052551)	0.086521 / 0.141683 (-0.055162)	1.478906 / 1.452155 (0.026752)	1.512777 / 1.492716 (0.020060)

Benchmark: benchmark_getitem_100B.json

metric	get_batch_of_1024_random_rows	get_batch_of_1024_rows	get_first_row	get_last_row
new / old (diff)	0.008767 / 0.018006 (-0.009239)	0.397386 / 0.000490 (0.396897)	0.003136 / 0.000200 (0.002936)	0.000096 / 0.000054 (0.000042)

Benchmark: benchmark_indices_mapping.json

metric	select	shard	shuffle	sort	train_test_split
new / old (diff)	0.022804 / 0.037411 (-0.014608)	0.097591 / 0.014526 (0.083066)	0.103189 / 0.176557 (-0.073368)	0.138165 / 0.737135 (-0.598970)	0.107464 / 0.296338 (-0.188874)

Benchmark: benchmark_iterating.json

metric	read 5000	read 50000	read_batch 50000 10	read_batch 50000 100	read_batch 50000 1000	read_formatted numpy 5000	read_formatted pandas 5000	read_formatted tensorflow 5000	read_formatted torch 5000	read_formatted_batch numpy 5000 10	read_formatted_batch numpy 5000 1000	shuffled read 5000	shuffled read 50000	shuffled read_batch 50000 10	shuffled read_batch 50000 100	shuffled read_batch 50000 1000	shuffled read_formatted numpy 5000	shuffled read_formatted_batch numpy 5000 10	shuffled read_formatted_batch numpy 5000 1000
new / old (diff)	0.428956 / 0.215209 (0.213747)	4.269656 / 2.077655 (2.192001)	2.154418 / 1.504120 (0.650298)	1.914176 / 1.541195 (0.372982)	1.818452 / 1.468490 (0.349962)	0.701381 / 4.584777 (-3.883396)	3.425190 / 3.745712 (-0.320522)	1.862545 / 5.269862 (-3.407316)	1.166271 / 4.565676 (-3.399405)	0.083678 / 0.424275 (-0.340597)	0.012254 / 0.007607 (0.004647)	0.535710 / 0.226044 (0.309665)	5.342528 / 2.268929 (3.073600)	2.627135 / 55.444624 (-52.817489)	2.308313 / 6.876477 (-4.568164)	2.325568 / 2.142072 (0.183496)	0.818318 / 4.805227 (-3.986909)	0.149812 / 6.500664 (-6.350853)	0.064559 / 0.075469 (-0.010910)

Benchmark: benchmark_map_filter.json

metric	filter	map fast-tokenizer batched	map identity	map identity batched	map no-op batched	map no-op batched numpy	map no-op batched pandas	map no-op batched pytorch	map no-op batched tensorflow
new / old (diff)	1.253611 / 1.841788 (-0.588176)	13.646763 / 8.074308 (5.572455)	14.387630 / 10.191392 (4.196238)	0.159937 / 0.680424 (-0.520487)	0.029123 / 0.534201 (-0.505078)	0.400909 / 0.579283 (-0.178374)	0.422830 / 0.434364 (-0.011534)	0.488205 / 0.540337 (-0.052133)	0.577982 / 1.386936 (-0.808954)

PyArrow==latest

Show updated benchmarks!

Benchmark: benchmark_array_xd.json

metric	read_batch_formatted_as_numpy after write_array2d	read_batch_formatted_as_numpy after write_flattened_sequence	read_batch_formatted_as_numpy after write_nested_sequence	read_batch_unformated after write_array2d	read_batch_unformated after write_flattened_sequence	read_batch_unformated after write_nested_sequence	read_col_formatted_as_numpy after write_array2d	read_col_formatted_as_numpy after write_flattened_sequence	read_col_formatted_as_numpy after write_nested_sequence	read_col_unformated after write_array2d	read_col_unformated after write_flattened_sequence	read_col_unformated after write_nested_sequence	read_formatted_as_numpy after write_array2d	read_formatted_as_numpy after write_flattened_sequence	read_formatted_as_numpy after write_nested_sequence	read_unformated after write_array2d	read_unformated after write_flattened_sequence	read_unformated after write_nested_sequence	write_array2d	write_flattened_sequence	write_nested_sequence
new / old (diff)	0.006430 / 0.011353 (-0.004923)	0.004433 / 0.011008 (-0.006576)	0.077459 / 0.038508 (0.038951)	0.026949 / 0.023109 (0.003840)	0.350276 / 0.275898 (0.074378)	0.376189 / 0.323480 (0.052709)	0.004945 / 0.007986 (-0.003041)	0.003280 / 0.004328 (-0.001048)	0.076465 / 0.004250 (0.072215)	0.037510 / 0.037052 (0.000457)	0.350410 / 0.258489 (0.091921)	0.386778 / 0.293841 (0.092937)	0.031933 / 0.128546 (-0.096613)	0.011691 / 0.075646 (-0.063956)	0.086519 / 0.419271 (-0.332753)	0.042490 / 0.043533 (-0.001043)	0.355930 / 0.255139 (0.100791)	0.366500 / 0.283200 (0.083301)	0.089542 / 0.141683 (-0.052141)	1.492859 / 1.452155 (0.040704)	1.548626 / 1.492716 (0.055910)

Benchmark: benchmark_getitem_100B.json

metric	get_batch_of_1024_random_rows	get_batch_of_1024_rows	get_first_row	get_last_row
new / old (diff)	0.220123 / 0.018006 (0.202117)	0.396970 / 0.000490 (0.396480)	0.000398 / 0.000200 (0.000198)	0.000059 / 0.000054 (0.000004)

Benchmark: benchmark_indices_mapping.json

metric	select	shard	shuffle	sort	train_test_split
new / old (diff)	0.024831 / 0.037411 (-0.012580)	0.099681 / 0.014526 (0.085156)	0.108922 / 0.176557 (-0.067635)	0.143004 / 0.737135 (-0.594131)	0.109671 / 0.296338 (-0.186667)

Benchmark: benchmark_iterating.json

metric	read 5000	read 50000	read_batch 50000 10	read_batch 50000 100	read_batch 50000 1000	read_formatted numpy 5000	read_formatted pandas 5000	read_formatted tensorflow 5000	read_formatted torch 5000	read_formatted_batch numpy 5000 10	read_formatted_batch numpy 5000 1000	shuffled read 5000	shuffled read 50000	shuffled read_batch 50000 10	shuffled read_batch 50000 100	shuffled read_batch 50000 1000	shuffled read_formatted numpy 5000	shuffled read_formatted_batch numpy 5000 10	shuffled read_formatted_batch numpy 5000 1000
new / old (diff)	0.444237 / 0.215209 (0.229028)	4.430330 / 2.077655 (2.352675)	2.235003 / 1.504120 (0.730883)	2.010499 / 1.541195 (0.469305)	2.030585 / 1.468490 (0.562095)	0.701938 / 4.584777 (-3.882839)	3.334569 / 3.745712 (-0.411144)	1.861680 / 5.269862 (-3.408181)	1.166072 / 4.565676 (-3.399604)	0.083870 / 0.424275 (-0.340405)	0.012615 / 0.007607 (0.005008)	0.548789 / 0.226044 (0.322744)	5.488064 / 2.268929 (3.219136)	2.614926 / 55.444624 (-52.829698)	2.246455 / 6.876477 (-4.630022)	2.277439 / 2.142072 (0.135367)	0.808449 / 4.805227 (-3.996778)	0.152434 / 6.500664 (-6.348230)	0.066709 / 0.075469 (-0.008760)

Benchmark: benchmark_map_filter.json

metric	filter	map fast-tokenizer batched	map identity	map identity batched	map no-op batched	map no-op batched numpy	map no-op batched pandas	map no-op batched pytorch	map no-op batched tensorflow
new / old (diff)	1.316880 / 1.841788 (-0.524908)	13.965269 / 8.074308 (5.890961)	13.660187 / 10.191392 (3.468795)	0.157801 / 0.680424 (-0.522623)	0.016580 / 0.534201 (-0.517621)	0.382834 / 0.579283 (-0.196449)	0.394717 / 0.434364 (-0.039647)	0.465138 / 0.540337 (-0.075200)	0.552399 / 1.386936 (-0.834537)

[github-actions]

Show benchmarks

PyArrow==6.0.0

Show updated benchmarks!

Benchmark: benchmark_array_xd.json

metric	read_batch_formatted_as_numpy after write_array2d	read_batch_formatted_as_numpy after write_flattened_sequence	read_batch_formatted_as_numpy after write_nested_sequence	read_batch_unformated after write_array2d	read_batch_unformated after write_flattened_sequence	read_batch_unformated after write_nested_sequence	read_col_formatted_as_numpy after write_array2d	read_col_formatted_as_numpy after write_flattened_sequence	read_col_formatted_as_numpy after write_nested_sequence	read_col_unformated after write_array2d	read_col_unformated after write_flattened_sequence	read_col_unformated after write_nested_sequence	read_formatted_as_numpy after write_array2d	read_formatted_as_numpy after write_flattened_sequence	read_formatted_as_numpy after write_nested_sequence	read_unformated after write_array2d	read_unformated after write_flattened_sequence	read_unformated after write_nested_sequence	write_array2d	write_flattened_sequence	write_nested_sequence
new / old (diff)	0.009341 / 0.011353 (-0.002012)	0.005303 / 0.011008 (-0.005705)	0.099287 / 0.038508 (0.060779)	0.035587 / 0.023109 (0.012478)	0.295146 / 0.275898 (0.019248)	0.370470 / 0.323480 (0.046990)	0.008910 / 0.007986 (0.000925)	0.004358 / 0.004328 (0.000029)	0.076298 / 0.004250 (0.072047)	0.047187 / 0.037052 (0.010135)	0.309025 / 0.258489 (0.050536)	0.346659 / 0.293841 (0.052818)	0.038378 / 0.128546 (-0.090168)	0.012475 / 0.075646 (-0.063172)	0.334370 / 0.419271 (-0.084901)	0.048391 / 0.043533 (0.004858)	0.298613 / 0.255139 (0.043474)	0.317329 / 0.283200 (0.034130)	0.108748 / 0.141683 (-0.032934)	1.450454 / 1.452155 (-0.001701)	1.519883 / 1.492716 (0.027167)

Benchmark: benchmark_getitem_100B.json

metric	get_batch_of_1024_random_rows	get_batch_of_1024_rows	get_first_row	get_last_row
new / old (diff)	0.011513 / 0.018006 (-0.006494)	0.498941 / 0.000490 (0.498451)	0.005098 / 0.000200 (0.004898)	0.000096 / 0.000054 (0.000042)

Benchmark: benchmark_indices_mapping.json

metric	select	shard	shuffle	sort	train_test_split
new / old (diff)	0.030523 / 0.037411 (-0.006888)	0.105478 / 0.014526 (0.090952)	0.121101 / 0.176557 (-0.055456)	0.159951 / 0.737135 (-0.577184)	0.126766 / 0.296338 (-0.169572)

Benchmark: benchmark_iterating.json

metric	read 5000	read 50000	read_batch 50000 10	read_batch 50000 100	read_batch 50000 1000	read_formatted numpy 5000	read_formatted pandas 5000	read_formatted tensorflow 5000	read_formatted torch 5000	read_formatted_batch numpy 5000 10	read_formatted_batch numpy 5000 1000	shuffled read 5000	shuffled read 50000	shuffled read_batch 50000 10	shuffled read_batch 50000 100	shuffled read_batch 50000 1000	shuffled read_formatted numpy 5000	shuffled read_formatted_batch numpy 5000 10	shuffled read_formatted_batch numpy 5000 1000
new / old (diff)	0.399101 / 0.215209 (0.183892)	3.997069 / 2.077655 (1.919414)	1.851592 / 1.504120 (0.347472)	1.695708 / 1.541195 (0.154513)	1.759504 / 1.468490 (0.291014)	0.708241 / 4.584777 (-3.876536)	3.786724 / 3.745712 (0.041012)	3.523731 / 5.269862 (-1.746131)	1.899474 / 4.565676 (-2.666203)	0.086680 / 0.424275 (-0.337595)	0.012232 / 0.007607 (0.004625)	0.508507 / 0.226044 (0.282462)	5.086320 / 2.268929 (2.817391)	2.234906 / 55.444624 (-53.209718)	1.911090 / 6.876477 (-4.965386)	1.989232 / 2.142072 (-0.152841)	0.863660 / 4.805227 (-3.941567)	0.169334 / 6.500664 (-6.331330)	0.063273 / 0.075469 (-0.012196)

Benchmark: benchmark_map_filter.json

metric	filter	map fast-tokenizer batched	map identity	map identity batched	map no-op batched	map no-op batched numpy	map no-op batched pandas	map no-op batched pytorch	map no-op batched tensorflow
new / old (diff)	1.237590 / 1.841788 (-0.604198)	15.417631 / 8.074308 (7.343323)	15.235308 / 10.191392 (5.043916)	0.209431 / 0.680424 (-0.470993)	0.029214 / 0.534201 (-0.504987)	0.444767 / 0.579283 (-0.134516)	0.447776 / 0.434364 (0.013413)	0.538440 / 0.540337 (-0.001897)	0.635760 / 1.386936 (-0.751176)

PyArrow==latest

Show updated benchmarks!

Benchmark: benchmark_array_xd.json

metric	read_batch_formatted_as_numpy after write_array2d	read_batch_formatted_as_numpy after write_flattened_sequence	read_batch_formatted_as_numpy after write_nested_sequence	read_batch_unformated after write_array2d	read_batch_unformated after write_flattened_sequence	read_batch_unformated after write_nested_sequence	read_col_formatted_as_numpy after write_array2d	read_col_formatted_as_numpy after write_flattened_sequence	read_col_formatted_as_numpy after write_nested_sequence	read_col_unformated after write_array2d	read_col_unformated after write_flattened_sequence	read_col_unformated after write_nested_sequence	read_formatted_as_numpy after write_array2d	read_formatted_as_numpy after write_flattened_sequence	read_formatted_as_numpy after write_nested_sequence	read_unformated after write_array2d	read_unformated after write_flattened_sequence	read_unformated after write_nested_sequence	write_array2d	write_flattened_sequence	write_nested_sequence
new / old (diff)	0.007758 / 0.011353 (-0.003594)	0.005539 / 0.011008 (-0.005469)	0.077011 / 0.038508 (0.038503)	0.034305 / 0.023109 (0.011196)	0.363352 / 0.275898 (0.087454)	0.411882 / 0.323480 (0.088403)	0.006286 / 0.007986 (-0.001700)	0.004378 / 0.004328 (0.000050)	0.075504 / 0.004250 (0.071253)	0.052728 / 0.037052 (0.015675)	0.370122 / 0.258489 (0.111633)	0.421910 / 0.293841 (0.128069)	0.038444 / 0.128546 (-0.090102)	0.012602 / 0.075646 (-0.063045)	0.088540 / 0.419271 (-0.330731)	0.060321 / 0.043533 (0.016788)	0.350502 / 0.255139 (0.095363)	0.393211 / 0.283200 (0.110011)	0.113057 / 0.141683 (-0.028626)	1.453275 / 1.452155 (0.001120)	1.541033 / 1.492716 (0.048317)

Benchmark: benchmark_getitem_100B.json

metric	get_batch_of_1024_random_rows	get_batch_of_1024_rows	get_first_row	get_last_row
new / old (diff)	0.333603 / 0.018006 (0.315597)	0.510548 / 0.000490 (0.510058)	0.003573 / 0.000200 (0.003373)	0.000116 / 0.000054 (0.000061)

Benchmark: benchmark_indices_mapping.json

metric	select	shard	shuffle	sort	train_test_split
new / old (diff)	0.032783 / 0.037411 (-0.004628)	0.111943 / 0.014526 (0.097418)	0.127154 / 0.176557 (-0.049403)	0.171716 / 0.737135 (-0.565420)	0.132441 / 0.296338 (-0.163898)

Benchmark: benchmark_iterating.json

metric	read 5000	read 50000	read_batch 50000 10	read_batch 50000 100	read_batch 50000 1000	read_formatted numpy 5000	read_formatted pandas 5000	read_formatted tensorflow 5000	read_formatted torch 5000	read_formatted_batch numpy 5000 10	read_formatted_batch numpy 5000 1000	shuffled read 5000	shuffled read 50000	shuffled read_batch 50000 10	shuffled read_batch 50000 100	shuffled read_batch 50000 1000	shuffled read_formatted numpy 5000	shuffled read_formatted_batch numpy 5000 10	shuffled read_formatted_batch numpy 5000 1000
new / old (diff)	0.439110 / 0.215209 (0.223901)	4.440874 / 2.077655 (2.363220)	2.145850 / 1.504120 (0.641730)	1.909566 / 1.541195 (0.368371)	2.032199 / 1.468490 (0.563709)	0.711295 / 4.584777 (-3.873482)	3.845729 / 3.745712 (0.100017)	3.583555 / 5.269862 (-1.686307)	1.836856 / 4.565676 (-2.728820)	0.085966 / 0.424275 (-0.338309)	0.012479 / 0.007607 (0.004872)	0.545379 / 0.226044 (0.319334)	5.425724 / 2.268929 (3.156796)	2.648304 / 55.444624 (-52.796321)	2.286369 / 6.876477 (-4.590108)	2.367714 / 2.142072 (0.225642)	0.831035 / 4.805227 (-3.974192)	0.167603 / 6.500664 (-6.333061)	0.064721 / 0.075469 (-0.010748)

Benchmark: benchmark_map_filter.json

metric	filter	map fast-tokenizer batched	map identity	map identity batched	map no-op batched	map no-op batched numpy	map no-op batched pandas	map no-op batched pytorch	map no-op batched tensorflow
new / old (diff)	1.244495 / 1.841788 (-0.597292)	15.304267 / 8.074308 (7.229958)	13.912185 / 10.191392 (3.720793)	0.156459 / 0.680424 (-0.523965)	0.019181 / 0.534201 (-0.515019)	0.425940 / 0.579283 (-0.153343)	0.427956 / 0.434364 (-0.006408)	0.529126 / 0.540337 (-0.011212)	0.628360 / 1.386936 (-0.758576)