Document that removing all the columns returns an empty document and the num_row is lost

[thomasw21]

No description provided.

[HuggingFaceDocBuilderDev]

The documentation is not available anymore as the PR was closed or merged.

[mariosasko]

Nit

[github-actions]

Show benchmarks

PyArrow==6.0.0

Show updated benchmarks!

Benchmark: benchmark_array_xd.json

metric	read_batch_formatted_as_numpy after write_array2d	read_batch_formatted_as_numpy after write_flattened_sequence	read_batch_formatted_as_numpy after write_nested_sequence	read_batch_unformated after write_array2d	read_batch_unformated after write_flattened_sequence	read_batch_unformated after write_nested_sequence	read_col_formatted_as_numpy after write_array2d	read_col_formatted_as_numpy after write_flattened_sequence	read_col_formatted_as_numpy after write_nested_sequence	read_col_unformated after write_array2d	read_col_unformated after write_flattened_sequence	read_col_unformated after write_nested_sequence	read_formatted_as_numpy after write_array2d	read_formatted_as_numpy after write_flattened_sequence	read_formatted_as_numpy after write_nested_sequence	read_unformated after write_array2d	read_unformated after write_flattened_sequence	read_unformated after write_nested_sequence	write_array2d	write_flattened_sequence	write_nested_sequence
new / old (diff)	0.011812 / 0.011353 (0.000459)	0.006878 / 0.011008 (-0.004130)	0.128720 / 0.038508 (0.090212)	0.038506 / 0.023109 (0.015397)	0.359670 / 0.275898 (0.083772)	0.422908 / 0.323480 (0.099428)	0.010115 / 0.007986 (0.002129)	0.004332 / 0.004328 (0.000004)	0.096281 / 0.004250 (0.092031)	0.048850 / 0.037052 (0.011798)	0.373795 / 0.258489 (0.115306)	0.414643 / 0.293841 (0.120802)	0.057568 / 0.128546 (-0.070978)	0.024135 / 0.075646 (-0.051512)	0.411764 / 0.419271 (-0.007507)	0.060167 / 0.043533 (0.016634)	0.367119 / 0.255139 (0.111980)	0.391813 / 0.283200 (0.108613)	0.112125 / 0.141683 (-0.029558)	1.869560 / 1.452155 (0.417406)	1.845649 / 1.492716 (0.352932)

Benchmark: benchmark_getitem_100B.json

metric	get_batch_of_1024_random_rows	get_batch_of_1024_rows	get_first_row	get_last_row
new / old (diff)	0.211449 / 0.018006 (0.193443)	0.522453 / 0.000490 (0.521963)	0.003984 / 0.000200 (0.003784)	0.000096 / 0.000054 (0.000042)

Benchmark: benchmark_indices_mapping.json

metric	select	shard	shuffle	sort	train_test_split
new / old (diff)	0.026015 / 0.037411 (-0.011397)	0.117747 / 0.014526 (0.103221)	0.125037 / 0.176557 (-0.051520)	0.168351 / 0.737135 (-0.568785)	0.132390 / 0.296338 (-0.163949)

Benchmark: benchmark_iterating.json

metric	read 5000	read 50000	read_batch 50000 10	read_batch 50000 100	read_batch 50000 1000	read_formatted numpy 5000	read_formatted pandas 5000	read_formatted tensorflow 5000	read_formatted torch 5000	read_formatted_batch numpy 5000 10	read_formatted_batch numpy 5000 1000	shuffled read 5000	shuffled read 50000	shuffled read_batch 50000 10	shuffled read_batch 50000 100	shuffled read_batch 50000 1000	shuffled read_formatted numpy 5000	shuffled read_formatted_batch numpy 5000 10	shuffled read_formatted_batch numpy 5000 1000
new / old (diff)	0.605653 / 0.215209 (0.390444)	5.883452 / 2.077655 (3.805798)	2.367052 / 1.504120 (0.862932)	2.137671 / 1.541195 (0.596476)	2.042370 / 1.468490 (0.573880)	1.168442 / 4.584777 (-3.416335)	5.205236 / 3.745712 (1.459524)	2.992514 / 5.269862 (-2.277348)	2.191829 / 4.565676 (-2.373847)	0.137702 / 0.424275 (-0.286574)	0.015898 / 0.007607 (0.008291)	0.783987 / 0.226044 (0.557942)	7.768965 / 2.268929 (5.500036)	3.249149 / 55.444624 (-52.195476)	2.530687 / 6.876477 (-4.345790)	2.675212 / 2.142072 (0.533140)	1.482804 / 4.805227 (-3.322423)	0.276845 / 6.500664 (-6.223819)	0.080597 / 0.075469 (0.005128)

Benchmark: benchmark_map_filter.json

metric	filter	map fast-tokenizer batched	map identity	map identity batched	map no-op batched	map no-op batched numpy	map no-op batched pandas	map no-op batched pytorch	map no-op batched tensorflow
new / old (diff)	1.519086 / 1.841788 (-0.322701)	17.394093 / 8.074308 (9.319785)	19.613554 / 10.191392 (9.422162)	0.253291 / 0.680424 (-0.427133)	0.047746 / 0.534201 (-0.486455)	0.547114 / 0.579283 (-0.032170)	0.623873 / 0.434364 (0.189509)	0.631924 / 0.540337 (0.091586)	0.744390 / 1.386936 (-0.642546)

PyArrow==latest

Show updated benchmarks!

Benchmark: benchmark_array_xd.json

metric	read_batch_formatted_as_numpy after write_array2d	read_batch_formatted_as_numpy after write_flattened_sequence	read_batch_formatted_as_numpy after write_nested_sequence	read_batch_unformated after write_array2d	read_batch_unformated after write_flattened_sequence	read_batch_unformated after write_nested_sequence	read_col_formatted_as_numpy after write_array2d	read_col_formatted_as_numpy after write_flattened_sequence	read_col_formatted_as_numpy after write_nested_sequence	read_col_unformated after write_array2d	read_col_unformated after write_flattened_sequence	read_col_unformated after write_nested_sequence	read_formatted_as_numpy after write_array2d	read_formatted_as_numpy after write_flattened_sequence	read_formatted_as_numpy after write_nested_sequence	read_unformated after write_array2d	read_unformated after write_flattened_sequence	read_unformated after write_nested_sequence	write_array2d	write_flattened_sequence	write_nested_sequence
new / old (diff)	0.009229 / 0.011353 (-0.002124)	0.006206 / 0.011008 (-0.004802)	0.121866 / 0.038508 (0.083357)	0.033629 / 0.023109 (0.010519)	0.435172 / 0.275898 (0.159274)	0.472093 / 0.323480 (0.148613)	0.006946 / 0.007986 (-0.001039)	0.004848 / 0.004328 (0.000519)	0.097289 / 0.004250 (0.093038)	0.046982 / 0.037052 (0.009930)	0.447365 / 0.258489 (0.188876)	0.491213 / 0.293841 (0.197372)	0.055486 / 0.128546 (-0.073060)	0.019788 / 0.075646 (-0.055858)	0.399830 / 0.419271 (-0.019441)	0.058943 / 0.043533 (0.015411)	0.447658 / 0.255139 (0.192519)	0.465752 / 0.283200 (0.182552)	0.110441 / 0.141683 (-0.031242)	1.773155 / 1.452155 (0.321001)	1.899370 / 1.492716 (0.406653)

Benchmark: benchmark_getitem_100B.json

metric	get_batch_of_1024_random_rows	get_batch_of_1024_rows	get_first_row	get_last_row
new / old (diff)	0.191188 / 0.018006 (0.173181)	0.523721 / 0.000490 (0.523232)	0.004008 / 0.000200 (0.003808)	0.000126 / 0.000054 (0.000072)

Benchmark: benchmark_indices_mapping.json

metric	select	shard	shuffle	sort	train_test_split
new / old (diff)	0.032579 / 0.037411 (-0.004833)	0.120870 / 0.014526 (0.106344)	0.154991 / 0.176557 (-0.021565)	0.175450 / 0.737135 (-0.561685)	0.136526 / 0.296338 (-0.159813)

Benchmark: benchmark_iterating.json

metric	read 5000	read 50000	read_batch 50000 10	read_batch 50000 100	read_batch 50000 1000	read_formatted numpy 5000	read_formatted pandas 5000	read_formatted tensorflow 5000	read_formatted torch 5000	read_formatted_batch numpy 5000 10	read_formatted_batch numpy 5000 1000	shuffled read 5000	shuffled read 50000	shuffled read_batch 50000 10	shuffled read_batch 50000 100	shuffled read_batch 50000 1000	shuffled read_formatted numpy 5000	shuffled read_formatted_batch numpy 5000 10	shuffled read_formatted_batch numpy 5000 1000
new / old (diff)	0.627262 / 0.215209 (0.412052)	6.457989 / 2.077655 (4.380334)	2.935188 / 1.504120 (1.431068)	2.558705 / 1.541195 (1.017510)	2.669455 / 1.468490 (1.200965)	1.228791 / 4.584777 (-3.355985)	5.621262 / 3.745712 (1.875549)	3.181775 / 5.269862 (-2.088086)	2.115116 / 4.565676 (-2.450560)	0.159348 / 0.424275 (-0.264927)	0.013598 / 0.007607 (0.005991)	0.834732 / 0.226044 (0.608687)	8.051097 / 2.268929 (5.782168)	3.761681 / 55.444624 (-51.682943)	2.898158 / 6.876477 (-3.978319)	2.936289 / 2.142072 (0.794217)	1.476307 / 4.805227 (-3.328920)	0.269845 / 6.500664 (-6.230819)	0.087225 / 0.075469 (0.011756)

Benchmark: benchmark_map_filter.json

metric	filter	map fast-tokenizer batched	map identity	map identity batched	map no-op batched	map no-op batched numpy	map no-op batched pandas	map no-op batched pytorch	map no-op batched tensorflow
new / old (diff)	1.632522 / 1.841788 (-0.209266)	17.615297 / 8.074308 (9.540989)	20.501172 / 10.191392 (10.309780)	0.248845 / 0.680424 (-0.431579)	0.024852 / 0.534201 (-0.509349)	0.498957 / 0.579283 (-0.080326)	0.588566 / 0.434364 (0.154202)	0.611051 / 0.540337 (0.070714)	0.726321 / 1.386936 (-0.660615)

[github-actions]

Show benchmarks

PyArrow==6.0.0

Show updated benchmarks!

Benchmark: benchmark_array_xd.json

metric	read_batch_formatted_as_numpy after write_array2d	read_batch_formatted_as_numpy after write_flattened_sequence	read_batch_formatted_as_numpy after write_nested_sequence	read_batch_unformated after write_array2d	read_batch_unformated after write_flattened_sequence	read_batch_unformated after write_nested_sequence	read_col_formatted_as_numpy after write_array2d	read_col_formatted_as_numpy after write_flattened_sequence	read_col_formatted_as_numpy after write_nested_sequence	read_col_unformated after write_array2d	read_col_unformated after write_flattened_sequence	read_col_unformated after write_nested_sequence	read_formatted_as_numpy after write_array2d	read_formatted_as_numpy after write_flattened_sequence	read_formatted_as_numpy after write_nested_sequence	read_unformated after write_array2d	read_unformated after write_flattened_sequence	read_unformated after write_nested_sequence	write_array2d	write_flattened_sequence	write_nested_sequence
new / old (diff)	0.008920 / 0.011353 (-0.002433)	0.004666 / 0.011008 (-0.006342)	0.098584 / 0.038508 (0.060076)	0.030213 / 0.023109 (0.007103)	0.298180 / 0.275898 (0.022282)	0.358932 / 0.323480 (0.035452)	0.007182 / 0.007986 (-0.000804)	0.005430 / 0.004328 (0.001102)	0.077962 / 0.004250 (0.073712)	0.038516 / 0.037052 (0.001463)	0.308840 / 0.258489 (0.050351)	0.343678 / 0.293841 (0.049837)	0.033701 / 0.128546 (-0.094845)	0.011460 / 0.075646 (-0.064186)	0.319809 / 0.419271 (-0.099462)	0.040731 / 0.043533 (-0.002802)	0.299772 / 0.255139 (0.044633)	0.324292 / 0.283200 (0.041092)	0.087755 / 0.141683 (-0.053928)	1.493077 / 1.452155 (0.040922)	1.527462 / 1.492716 (0.034746)

Benchmark: benchmark_getitem_100B.json

metric	get_batch_of_1024_random_rows	get_batch_of_1024_rows	get_first_row	get_last_row
new / old (diff)	0.187927 / 0.018006 (0.169921)	0.412785 / 0.000490 (0.412296)	0.003235 / 0.000200 (0.003035)	0.000080 / 0.000054 (0.000026)

Benchmark: benchmark_indices_mapping.json

metric	select	shard	shuffle	sort	train_test_split
new / old (diff)	0.023313 / 0.037411 (-0.014098)	0.095663 / 0.014526 (0.081137)	0.105094 / 0.176557 (-0.071463)	0.140389 / 0.737135 (-0.596746)	0.108477 / 0.296338 (-0.187861)

Benchmark: benchmark_iterating.json

metric	read 5000	read 50000	read_batch 50000 10	read_batch 50000 100	read_batch 50000 1000	read_formatted numpy 5000	read_formatted pandas 5000	read_formatted tensorflow 5000	read_formatted torch 5000	read_formatted_batch numpy 5000 10	read_formatted_batch numpy 5000 1000	shuffled read 5000	shuffled read 50000	shuffled read_batch 50000 10	shuffled read_batch 50000 100	shuffled read_batch 50000 1000	shuffled read_formatted numpy 5000	shuffled read_formatted_batch numpy 5000 10	shuffled read_formatted_batch numpy 5000 1000
new / old (diff)	0.410680 / 0.215209 (0.195471)	4.109287 / 2.077655 (2.031632)	1.833214 / 1.504120 (0.329094)	1.622837 / 1.541195 (0.081642)	1.679899 / 1.468490 (0.211409)	0.686920 / 4.584777 (-3.897857)	3.463267 / 3.745712 (-0.282445)	1.867035 / 5.269862 (-3.402826)	1.150631 / 4.565676 (-3.415046)	0.081209 / 0.424275 (-0.343066)	0.012384 / 0.007607 (0.004777)	0.521070 / 0.226044 (0.295026)	5.208829 / 2.268929 (2.939900)	2.289032 / 55.444624 (-53.155592)	1.942976 / 6.876477 (-4.933501)	1.990660 / 2.142072 (-0.151413)	0.802976 / 4.805227 (-4.002252)	0.148199 / 6.500664 (-6.352465)	0.064644 / 0.075469 (-0.010825)

Benchmark: benchmark_map_filter.json

metric	filter	map fast-tokenizer batched	map identity	map identity batched	map no-op batched	map no-op batched numpy	map no-op batched pandas	map no-op batched pytorch	map no-op batched tensorflow
new / old (diff)	1.277029 / 1.841788 (-0.564759)	13.915489 / 8.074308 (5.841181)	14.035486 / 10.191392 (3.844094)	0.138205 / 0.680424 (-0.542219)	0.028968 / 0.534201 (-0.505232)	0.394275 / 0.579283 (-0.185008)	0.399967 / 0.434364 (-0.034397)	0.460595 / 0.540337 (-0.079742)	0.537625 / 1.386936 (-0.849311)

PyArrow==latest

Show updated benchmarks!

Benchmark: benchmark_array_xd.json

metric	read_batch_formatted_as_numpy after write_array2d	read_batch_formatted_as_numpy after write_flattened_sequence	read_batch_formatted_as_numpy after write_nested_sequence	read_batch_unformated after write_array2d	read_batch_unformated after write_flattened_sequence	read_batch_unformated after write_nested_sequence	read_col_formatted_as_numpy after write_array2d	read_col_formatted_as_numpy after write_flattened_sequence	read_col_formatted_as_numpy after write_nested_sequence	read_col_unformated after write_array2d	read_col_unformated after write_flattened_sequence	read_col_unformated after write_nested_sequence	read_formatted_as_numpy after write_array2d	read_formatted_as_numpy after write_flattened_sequence	read_formatted_as_numpy after write_nested_sequence	read_unformated after write_array2d	read_unformated after write_flattened_sequence	read_unformated after write_nested_sequence	write_array2d	write_flattened_sequence	write_nested_sequence
new / old (diff)	0.006485 / 0.011353 (-0.004868)	0.004534 / 0.011008 (-0.006474)	0.097742 / 0.038508 (0.059234)	0.027231 / 0.023109 (0.004122)	0.431321 / 0.275898 (0.155423)	0.469212 / 0.323480 (0.145732)	0.004894 / 0.007986 (-0.003092)	0.004147 / 0.004328 (-0.000181)	0.073650 / 0.004250 (0.069400)	0.037052 / 0.037052 (-0.000000)	0.434196 / 0.258489 (0.175707)	0.480539 / 0.293841 (0.186698)	0.031923 / 0.128546 (-0.096623)	0.011522 / 0.075646 (-0.064124)	0.317062 / 0.419271 (-0.102209)	0.041124 / 0.043533 (-0.002409)	0.432013 / 0.255139 (0.176874)	0.456760 / 0.283200 (0.173560)	0.089757 / 0.141683 (-0.051925)	1.497752 / 1.452155 (0.045597)	1.585342 / 1.492716 (0.092626)

Benchmark: benchmark_getitem_100B.json

metric	get_batch_of_1024_random_rows	get_batch_of_1024_rows	get_first_row	get_last_row
new / old (diff)	0.227784 / 0.018006 (0.209778)	0.404570 / 0.000490 (0.404080)	0.000556 / 0.000200 (0.000356)	0.000065 / 0.000054 (0.000011)

Benchmark: benchmark_indices_mapping.json

metric	select	shard	shuffle	sort	train_test_split
new / old (diff)	0.025201 / 0.037411 (-0.012210)	0.099348 / 0.014526 (0.084822)	0.114984 / 0.176557 (-0.061573)	0.147039 / 0.737135 (-0.590097)	0.109727 / 0.296338 (-0.186611)

Benchmark: benchmark_iterating.json

metric	read 5000	read 50000	read_batch 50000 10	read_batch 50000 100	read_batch 50000 1000	read_formatted numpy 5000	read_formatted pandas 5000	read_formatted tensorflow 5000	read_formatted torch 5000	read_formatted_batch numpy 5000 10	read_formatted_batch numpy 5000 1000	shuffled read 5000	shuffled read 50000	shuffled read_batch 50000 10	shuffled read_batch 50000 100	shuffled read_batch 50000 1000	shuffled read_formatted numpy 5000	shuffled read_formatted_batch numpy 5000 10	shuffled read_formatted_batch numpy 5000 1000
new / old (diff)	0.468415 / 0.215209 (0.253206)	4.692228 / 2.077655 (2.614573)	2.403382 / 1.504120 (0.899262)	2.196026 / 1.541195 (0.654832)	2.234736 / 1.468490 (0.766246)	0.703011 / 4.584777 (-3.881766)	3.451513 / 3.745712 (-0.294199)	2.596811 / 5.269862 (-2.673051)	1.544079 / 4.565676 (-3.021598)	0.083153 / 0.424275 (-0.341123)	0.012605 / 0.007607 (0.004998)	0.570265 / 0.226044 (0.344220)	5.735996 / 2.268929 (3.467067)	2.865336 / 55.444624 (-52.579288)	2.508340 / 6.876477 (-4.368137)	2.547144 / 2.142072 (0.405072)	0.813018 / 4.805227 (-3.992210)	0.150327 / 6.500664 (-6.350337)	0.065837 / 0.075469 (-0.009632)

Benchmark: benchmark_map_filter.json

metric	filter	map fast-tokenizer batched	map identity	map identity batched	map no-op batched	map no-op batched numpy	map no-op batched pandas	map no-op batched pytorch	map no-op batched tensorflow
new / old (diff)	1.268941 / 1.841788 (-0.572847)	13.835698 / 8.074308 (5.761390)	13.992726 / 10.191392 (3.801334)	0.127751 / 0.680424 (-0.552673)	0.016673 / 0.534201 (-0.517528)	0.381921 / 0.579283 (-0.197362)	0.390688 / 0.434364 (-0.043676)	0.446234 / 0.540337 (-0.094103)	0.532631 / 1.386936 (-0.854305)

[mariosasko]

Thanks! The CI failures are unrelated to the changes, so feel free to merge

[github-actions]

Show benchmarks

PyArrow==6.0.0

Show updated benchmarks!

Benchmark: benchmark_array_xd.json

metric	read_batch_formatted_as_numpy after write_array2d	read_batch_formatted_as_numpy after write_flattened_sequence	read_batch_formatted_as_numpy after write_nested_sequence	read_batch_unformated after write_array2d	read_batch_unformated after write_flattened_sequence	read_batch_unformated after write_nested_sequence	read_col_formatted_as_numpy after write_array2d	read_col_formatted_as_numpy after write_flattened_sequence	read_col_formatted_as_numpy after write_nested_sequence	read_col_unformated after write_array2d	read_col_unformated after write_flattened_sequence	read_col_unformated after write_nested_sequence	read_formatted_as_numpy after write_array2d	read_formatted_as_numpy after write_flattened_sequence	read_formatted_as_numpy after write_nested_sequence	read_unformated after write_array2d	read_unformated after write_flattened_sequence	read_unformated after write_nested_sequence	write_array2d	write_flattened_sequence	write_nested_sequence
new / old (diff)	0.008486 / 0.011353 (-0.002867)	0.004573 / 0.011008 (-0.006435)	0.100096 / 0.038508 (0.061588)	0.029449 / 0.023109 (0.006340)	0.298384 / 0.275898 (0.022486)	0.361886 / 0.323480 (0.038406)	0.006813 / 0.007986 (-0.001173)	0.003394 / 0.004328 (-0.000935)	0.077563 / 0.004250 (0.073312)	0.035605 / 0.037052 (-0.001447)	0.306864 / 0.258489 (0.048375)	0.346438 / 0.293841 (0.052597)	0.033156 / 0.128546 (-0.095390)	0.011567 / 0.075646 (-0.064079)	0.322189 / 0.419271 (-0.097083)	0.040161 / 0.043533 (-0.003372)	0.299329 / 0.255139 (0.044190)	0.326375 / 0.283200 (0.043175)	0.086572 / 0.141683 (-0.055111)	1.502473 / 1.452155 (0.050319)	1.528539 / 1.492716 (0.035823)

Benchmark: benchmark_getitem_100B.json

metric	get_batch_of_1024_random_rows	get_batch_of_1024_rows	get_first_row	get_last_row
new / old (diff)	0.008502 / 0.018006 (-0.009505)	0.411045 / 0.000490 (0.410555)	0.003179 / 0.000200 (0.002980)	0.000073 / 0.000054 (0.000018)

Benchmark: benchmark_indices_mapping.json

metric	select	shard	shuffle	sort	train_test_split
new / old (diff)	0.023177 / 0.037411 (-0.014234)	0.096948 / 0.014526 (0.082422)	0.104068 / 0.176557 (-0.072489)	0.138739 / 0.737135 (-0.598396)	0.108241 / 0.296338 (-0.188097)

Benchmark: benchmark_iterating.json

metric	read 5000	read 50000	read_batch 50000 10	read_batch 50000 100	read_batch 50000 1000	read_formatted numpy 5000	read_formatted pandas 5000	read_formatted tensorflow 5000	read_formatted torch 5000	read_formatted_batch numpy 5000 10	read_formatted_batch numpy 5000 1000	shuffled read 5000	shuffled read 50000	shuffled read_batch 50000 10	shuffled read_batch 50000 100	shuffled read_batch 50000 1000	shuffled read_formatted numpy 5000	shuffled read_formatted_batch numpy 5000 10	shuffled read_formatted_batch numpy 5000 1000
new / old (diff)	0.411156 / 0.215209 (0.195947)	4.092992 / 2.077655 (2.015337)	1.841903 / 1.504120 (0.337783)	1.637449 / 1.541195 (0.096254)	1.670968 / 1.468490 (0.202478)	0.697301 / 4.584777 (-3.887476)	3.354717 / 3.745712 (-0.390995)	1.851518 / 5.269862 (-3.418344)	1.160367 / 4.565676 (-3.405309)	0.082613 / 0.424275 (-0.341662)	0.012477 / 0.007607 (0.004870)	0.524839 / 0.226044 (0.298795)	5.264173 / 2.268929 (2.995245)	2.294530 / 55.444624 (-53.150094)	1.933233 / 6.876477 (-4.943244)	1.968959 / 2.142072 (-0.173113)	0.817104 / 4.805227 (-3.988123)	0.149072 / 6.500664 (-6.351592)	0.064911 / 0.075469 (-0.010558)

Benchmark: benchmark_map_filter.json

metric	filter	map fast-tokenizer batched	map identity	map identity batched	map no-op batched	map no-op batched numpy	map no-op batched pandas	map no-op batched pytorch	map no-op batched tensorflow
new / old (diff)	1.222215 / 1.841788 (-0.619573)	13.607545 / 8.074308 (5.533237)	13.990230 / 10.191392 (3.798838)	0.150855 / 0.680424 (-0.529568)	0.028844 / 0.534201 (-0.505357)	0.396169 / 0.579283 (-0.183114)	0.406957 / 0.434364 (-0.027407)	0.464069 / 0.540337 (-0.076268)	0.554027 / 1.386936 (-0.832909)

PyArrow==latest

Show updated benchmarks!

Benchmark: benchmark_array_xd.json

metric	read_batch_formatted_as_numpy after write_array2d	read_batch_formatted_as_numpy after write_flattened_sequence	read_batch_formatted_as_numpy after write_nested_sequence	read_batch_unformated after write_array2d	read_batch_unformated after write_flattened_sequence	read_batch_unformated after write_nested_sequence	read_col_formatted_as_numpy after write_array2d	read_col_formatted_as_numpy after write_flattened_sequence	read_col_formatted_as_numpy after write_nested_sequence	read_col_unformated after write_array2d	read_col_unformated after write_flattened_sequence	read_col_unformated after write_nested_sequence	read_formatted_as_numpy after write_array2d	read_formatted_as_numpy after write_flattened_sequence	read_formatted_as_numpy after write_nested_sequence	read_unformated after write_array2d	read_unformated after write_flattened_sequence	read_unformated after write_nested_sequence	write_array2d	write_flattened_sequence	write_nested_sequence
new / old (diff)	0.006296 / 0.011353 (-0.005057)	0.004563 / 0.011008 (-0.006445)	0.097719 / 0.038508 (0.059211)	0.027106 / 0.023109 (0.003996)	0.409333 / 0.275898 (0.133435)	0.445397 / 0.323480 (0.121917)	0.004906 / 0.007986 (-0.003080)	0.003316 / 0.004328 (-0.001012)	0.075363 / 0.004250 (0.071112)	0.039366 / 0.037052 (0.002314)	0.412710 / 0.258489 (0.154221)	0.451789 / 0.293841 (0.157948)	0.031810 / 0.128546 (-0.096736)	0.011681 / 0.075646 (-0.063965)	0.318484 / 0.419271 (-0.100788)	0.046741 / 0.043533 (0.003208)	0.411631 / 0.255139 (0.156492)	0.435274 / 0.283200 (0.152074)	0.092366 / 0.141683 (-0.049317)	1.492243 / 1.452155 (0.040089)	1.617603 / 1.492716 (0.124887)

Benchmark: benchmark_getitem_100B.json

metric	get_batch_of_1024_random_rows	get_batch_of_1024_rows	get_first_row	get_last_row
new / old (diff)	0.217376 / 0.018006 (0.199369)	0.400940 / 0.000490 (0.400450)	0.003700 / 0.000200 (0.003500)	0.000075 / 0.000054 (0.000021)

Benchmark: benchmark_indices_mapping.json

metric	select	shard	shuffle	sort	train_test_split
new / old (diff)	0.023733 / 0.037411 (-0.013678)	0.098553 / 0.014526 (0.084027)	0.105790 / 0.176557 (-0.070767)	0.139537 / 0.737135 (-0.597598)	0.109862 / 0.296338 (-0.186477)

Benchmark: benchmark_iterating.json

metric	read 5000	read 50000	read_batch 50000 10	read_batch 50000 100	read_batch 50000 1000	read_formatted numpy 5000	read_formatted pandas 5000	read_formatted tensorflow 5000	read_formatted torch 5000	read_formatted_batch numpy 5000 10	read_formatted_batch numpy 5000 1000	shuffled read 5000	shuffled read 50000	shuffled read_batch 50000 10	shuffled read_batch 50000 100	shuffled read_batch 50000 1000	shuffled read_formatted numpy 5000	shuffled read_formatted_batch numpy 5000 10	shuffled read_formatted_batch numpy 5000 1000
new / old (diff)	0.476562 / 0.215209 (0.261353)	4.773469 / 2.077655 (2.695814)	2.447302 / 1.504120 (0.943182)	2.240596 / 1.541195 (0.699401)	2.271370 / 1.468490 (0.802880)	0.698913 / 4.584777 (-3.885864)	3.345648 / 3.745712 (-0.400064)	1.845008 / 5.269862 (-3.424854)	1.163213 / 4.565676 (-3.402464)	0.082456 / 0.424275 (-0.341819)	0.012315 / 0.007607 (0.004708)	0.575881 / 0.226044 (0.349836)	5.769575 / 2.268929 (3.500647)	2.909759 / 55.444624 (-52.534865)	2.580259 / 6.876477 (-4.296218)	2.590473 / 2.142072 (0.448401)	0.802765 / 4.805227 (-4.002462)	0.151514 / 6.500664 (-6.349150)	0.067718 / 0.075469 (-0.007751)

Benchmark: benchmark_map_filter.json

metric	filter	map fast-tokenizer batched	map identity	map identity batched	map no-op batched	map no-op batched numpy	map no-op batched pandas	map no-op batched pytorch	map no-op batched tensorflow
new / old (diff)	1.293014 / 1.841788 (-0.548773)	13.934072 / 8.074308 (5.859763)	13.538760 / 10.191392 (3.347368)	0.126490 / 0.680424 (-0.553934)	0.016653 / 0.534201 (-0.517548)	0.381220 / 0.579283 (-0.198064)	0.387571 / 0.434364 (-0.046793)	0.444674 / 0.540337 (-0.095663)	0.550802 / 1.386936 (-0.836134)