Improve features decoding in to_iterable_dataset

[lhoestq]

Following discussion at #5589

Right now to_iterable_dataset on images/audio hurts iterable dataset performance a lot (e.g. x4 slower because it encodes+decodes images/audios unnecessarily).

I fixed it by providing a generator that yields undecoded examples

[HuggingFaceDocBuilderDev]

The documentation is not available anymore as the PR was closed or merged.

[github-actions]

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

Benchmark: benchmark_array_xd.json

metric	read_batch_formatted_as_numpy after write_array2d	read_batch_formatted_as_numpy after write_flattened_sequence	read_batch_formatted_as_numpy after write_nested_sequence	read_batch_unformated after write_array2d	read_batch_unformated after write_flattened_sequence	read_batch_unformated after write_nested_sequence	read_col_formatted_as_numpy after write_array2d	read_col_formatted_as_numpy after write_flattened_sequence	read_col_formatted_as_numpy after write_nested_sequence	read_col_unformated after write_array2d	read_col_unformated after write_flattened_sequence	read_col_unformated after write_nested_sequence	read_formatted_as_numpy after write_array2d	read_formatted_as_numpy after write_flattened_sequence	read_formatted_as_numpy after write_nested_sequence	read_unformated after write_array2d	read_unformated after write_flattened_sequence	read_unformated after write_nested_sequence	write_array2d	write_flattened_sequence	write_nested_sequence
new / old (diff)	0.009691 / 0.011353 (-0.001662)	0.006160 / 0.011008 (-0.004848)	0.127528 / 0.038508 (0.089020)	0.034445 / 0.023109 (0.011335)	0.391483 / 0.275898 (0.115585)	0.425922 / 0.323480 (0.102442)	0.006621 / 0.007986 (-0.001365)	0.004550 / 0.004328 (0.000221)	0.099134 / 0.004250 (0.094884)	0.051089 / 0.037052 (0.014037)	0.398675 / 0.258489 (0.140186)	0.456740 / 0.293841 (0.162899)	0.052279 / 0.128546 (-0.076267)	0.020878 / 0.075646 (-0.054768)	0.414954 / 0.419271 (-0.004317)	0.061903 / 0.043533 (0.018370)	0.393088 / 0.255139 (0.137949)	0.410289 / 0.283200 (0.127089)	0.101684 / 0.141683 (-0.039998)	1.747102 / 1.452155 (0.294947)	1.896976 / 1.492716 (0.404260)

Benchmark: benchmark_getitem_100B.json

metric	get_batch_of_1024_random_rows	get_batch_of_1024_rows	get_first_row	get_last_row
new / old (diff)	0.203193 / 0.018006 (0.185187)	0.495011 / 0.000490 (0.494521)	0.006290 / 0.000200 (0.006090)	0.000098 / 0.000054 (0.000043)

Benchmark: benchmark_indices_mapping.json

metric	select	shard	shuffle	sort	train_test_split
new / old (diff)	0.034840 / 0.037411 (-0.002571)	0.122529 / 0.014526 (0.108003)	0.133870 / 0.176557 (-0.042686)	0.207771 / 0.737135 (-0.529364)	0.141441 / 0.296338 (-0.154897)

Benchmark: benchmark_iterating.json

metric	read 5000	read 50000	read_batch 50000 10	read_batch 50000 100	read_batch 50000 1000	read_formatted numpy 5000	read_formatted pandas 5000	read_formatted tensorflow 5000	read_formatted torch 5000	read_formatted_batch numpy 5000 10	read_formatted_batch numpy 5000 1000	shuffled read 5000	shuffled read 50000	shuffled read_batch 50000 10	shuffled read_batch 50000 100	shuffled read_batch 50000 1000	shuffled read_formatted numpy 5000	shuffled read_formatted_batch numpy 5000 10	shuffled read_formatted_batch numpy 5000 1000
new / old (diff)	0.604190 / 0.215209 (0.388981)	6.040295 / 2.077655 (3.962641)	2.405703 / 1.504120 (0.901583)	2.062767 / 1.541195 (0.521572)	2.079313 / 1.468490 (0.610823)	1.240107 / 4.584777 (-3.344670)	5.316583 / 3.745712 (1.570871)	3.104758 / 5.269862 (-2.165103)	2.056489 / 4.565676 (-2.509187)	0.149060 / 0.424275 (-0.275215)	0.014467 / 0.007607 (0.006860)	0.736882 / 0.226044 (0.510838)	7.324142 / 2.268929 (5.055213)	3.048752 / 55.444624 (-52.395872)	2.385013 / 6.876477 (-4.491463)	2.457478 / 2.142072 (0.315405)	1.459276 / 4.805227 (-3.345951)	0.253882 / 6.500664 (-6.246782)	0.076756 / 0.075469 (0.001287)

Benchmark: benchmark_map_filter.json

metric	filter	map fast-tokenizer batched	map identity	map identity batched	map no-op batched	map no-op batched numpy	map no-op batched pandas	map no-op batched pytorch	map no-op batched tensorflow
new / old (diff)	1.499166 / 1.841788 (-0.342622)	17.294165 / 8.074308 (9.219857)	20.385668 / 10.191392 (10.194276)	0.254633 / 0.680424 (-0.425791)	0.026253 / 0.534201 (-0.507948)	0.532928 / 0.579283 (-0.046355)	0.606095 / 0.434364 (0.171731)	0.615025 / 0.540337 (0.074687)	0.728651 / 1.386936 (-0.658285)

PyArrow==latest

Show updated benchmarks!

Benchmark: benchmark_array_xd.json

metric	read_batch_formatted_as_numpy after write_array2d	read_batch_formatted_as_numpy after write_flattened_sequence	read_batch_formatted_as_numpy after write_nested_sequence	read_batch_unformated after write_array2d	read_batch_unformated after write_flattened_sequence	read_batch_unformated after write_nested_sequence	read_col_formatted_as_numpy after write_array2d	read_col_formatted_as_numpy after write_flattened_sequence	read_col_formatted_as_numpy after write_nested_sequence	read_col_unformated after write_array2d	read_col_unformated after write_flattened_sequence	read_col_unformated after write_nested_sequence	read_formatted_as_numpy after write_array2d	read_formatted_as_numpy after write_flattened_sequence	read_formatted_as_numpy after write_nested_sequence	read_unformated after write_array2d	read_unformated after write_flattened_sequence	read_unformated after write_nested_sequence	write_array2d	write_flattened_sequence	write_nested_sequence
new / old (diff)	0.009376 / 0.011353 (-0.001977)	0.005981 / 0.011008 (-0.005027)	0.109898 / 0.038508 (0.071390)	0.033746 / 0.023109 (0.010637)	0.410226 / 0.275898 (0.134328)	0.470606 / 0.323480 (0.147126)	0.006706 / 0.007986 (-0.001279)	0.004482 / 0.004328 (0.000153)	0.092280 / 0.004250 (0.088030)	0.047988 / 0.037052 (0.010935)	0.430628 / 0.258489 (0.172139)	0.480668 / 0.293841 (0.186827)	0.052099 / 0.128546 (-0.076447)	0.018743 / 0.075646 (-0.056903)	0.112204 / 0.419271 (-0.307068)	0.059838 / 0.043533 (0.016305)	0.418230 / 0.255139 (0.163091)	0.451568 / 0.283200 (0.168368)	0.107026 / 0.141683 (-0.034657)	1.708111 / 1.452155 (0.255956)	1.839268 / 1.492716 (0.346552)

Benchmark: benchmark_getitem_100B.json

metric	get_batch_of_1024_random_rows	get_batch_of_1024_rows	get_first_row	get_last_row
new / old (diff)	0.229558 / 0.018006 (0.211552)	0.488099 / 0.000490 (0.487609)	0.004643 / 0.000200 (0.004443)	0.000107 / 0.000054 (0.000053)

Benchmark: benchmark_indices_mapping.json

metric	select	shard	shuffle	sort	train_test_split
new / old (diff)	0.030461 / 0.037411 (-0.006951)	0.120993 / 0.014526 (0.106467)	0.130874 / 0.176557 (-0.045682)	0.193550 / 0.737135 (-0.543585)	0.138164 / 0.296338 (-0.158174)

Benchmark: benchmark_iterating.json

metric	read 5000	read 50000	read_batch 50000 10	read_batch 50000 100	read_batch 50000 1000	read_formatted numpy 5000	read_formatted pandas 5000	read_formatted tensorflow 5000	read_formatted torch 5000	read_formatted_batch numpy 5000 10	read_formatted_batch numpy 5000 1000	shuffled read 5000	shuffled read 50000	shuffled read_batch 50000 10	shuffled read_batch 50000 100	shuffled read_batch 50000 1000	shuffled read_formatted numpy 5000	shuffled read_formatted_batch numpy 5000 10	shuffled read_formatted_batch numpy 5000 1000
new / old (diff)	0.635709 / 0.215209 (0.420500)	6.225112 / 2.077655 (4.147457)	2.639584 / 1.504120 (1.135465)	2.254487 / 1.541195 (0.713293)	2.280478 / 1.468490 (0.811988)	1.205712 / 4.584777 (-3.379065)	5.367845 / 3.745712 (1.622133)	3.020207 / 5.269862 (-2.249655)	2.001897 / 4.565676 (-2.563779)	0.149582 / 0.424275 (-0.274693)	0.014867 / 0.007607 (0.007260)	0.759050 / 0.226044 (0.533006)	7.692969 / 2.268929 (5.424041)	3.274009 / 55.444624 (-52.170615)	2.635529 / 6.876477 (-4.240948)	2.672960 / 2.142072 (0.530888)	1.426487 / 4.805227 (-3.378740)	0.253368 / 6.500664 (-6.247296)	0.078650 / 0.075469 (0.003181)

Benchmark: benchmark_map_filter.json

metric	filter	map fast-tokenizer batched	map identity	map identity batched	map no-op batched	map no-op batched numpy	map no-op batched pandas	map no-op batched pytorch	map no-op batched tensorflow
new / old (diff)	1.620265 / 1.841788 (-0.221523)	17.674168 / 8.074308 (9.599860)	21.120528 / 10.191392 (10.929136)	0.244205 / 0.680424 (-0.436218)	0.029646 / 0.534201 (-0.504555)	0.510948 / 0.579283 (-0.068335)	0.586255 / 0.434364 (0.151891)	0.589286 / 0.540337 (0.048949)	0.736561 / 1.386936 (-0.650375)

[github-actions]

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

Benchmark: benchmark_array_xd.json

metric	read_batch_formatted_as_numpy after write_array2d	read_batch_formatted_as_numpy after write_flattened_sequence	read_batch_formatted_as_numpy after write_nested_sequence	read_batch_unformated after write_array2d	read_batch_unformated after write_flattened_sequence	read_batch_unformated after write_nested_sequence	read_col_formatted_as_numpy after write_array2d	read_col_formatted_as_numpy after write_flattened_sequence	read_col_formatted_as_numpy after write_nested_sequence	read_col_unformated after write_array2d	read_col_unformated after write_flattened_sequence	read_col_unformated after write_nested_sequence	read_formatted_as_numpy after write_array2d	read_formatted_as_numpy after write_flattened_sequence	read_formatted_as_numpy after write_nested_sequence	read_unformated after write_array2d	read_unformated after write_flattened_sequence	read_unformated after write_nested_sequence	write_array2d	write_flattened_sequence	write_nested_sequence
new / old (diff)	0.007778 / 0.011353 (-0.003575)	0.005432 / 0.011008 (-0.005577)	0.098776 / 0.038508 (0.060268)	0.035196 / 0.023109 (0.012087)	0.305646 / 0.275898 (0.029748)	0.342661 / 0.323480 (0.019181)	0.006513 / 0.007986 (-0.001472)	0.005897 / 0.004328 (0.001568)	0.075797 / 0.004250 (0.071547)	0.056060 / 0.037052 (0.019007)	0.306645 / 0.258489 (0.048156)	0.352447 / 0.293841 (0.058606)	0.037304 / 0.128546 (-0.091242)	0.012514 / 0.075646 (-0.063132)	0.334949 / 0.419271 (-0.084323)	0.051600 / 0.043533 (0.008067)	0.302302 / 0.255139 (0.047163)	0.322238 / 0.283200 (0.039038)	0.106896 / 0.141683 (-0.034787)	1.483163 / 1.452155 (0.031008)	1.587483 / 1.492716 (0.094767)

Benchmark: benchmark_getitem_100B.json

metric	get_batch_of_1024_random_rows	get_batch_of_1024_rows	get_first_row	get_last_row
new / old (diff)	0.292318 / 0.018006 (0.274312)	0.541541 / 0.000490 (0.541051)	0.008342 / 0.000200 (0.008142)	0.000339 / 0.000054 (0.000285)

Benchmark: benchmark_indices_mapping.json

metric	select	shard	shuffle	sort	train_test_split
new / old (diff)	0.028287 / 0.037411 (-0.009124)	0.107775 / 0.014526 (0.093250)	0.119112 / 0.176557 (-0.057445)	0.174002 / 0.737135 (-0.563134)	0.126531 / 0.296338 (-0.169808)

Benchmark: benchmark_iterating.json

metric	read 5000	read 50000	read_batch 50000 10	read_batch 50000 100	read_batch 50000 1000	read_formatted numpy 5000	read_formatted pandas 5000	read_formatted tensorflow 5000	read_formatted torch 5000	read_formatted_batch numpy 5000 10	read_formatted_batch numpy 5000 1000	shuffled read 5000	shuffled read 50000	shuffled read_batch 50000 10	shuffled read_batch 50000 100	shuffled read_batch 50000 1000	shuffled read_formatted numpy 5000	shuffled read_formatted_batch numpy 5000 10	shuffled read_formatted_batch numpy 5000 1000
new / old (diff)	0.401684 / 0.215209 (0.186475)	4.024708 / 2.077655 (1.947053)	1.812763 / 1.504120 (0.308643)	1.629540 / 1.541195 (0.088345)	1.731733 / 1.468490 (0.263243)	0.711066 / 4.584777 (-3.873711)	3.867499 / 3.745712 (0.121786)	3.615968 / 5.269862 (-1.653893)	1.876077 / 4.565676 (-2.689600)	0.087003 / 0.424275 (-0.337272)	0.012445 / 0.007607 (0.004838)	0.499106 / 0.226044 (0.273061)	4.975920 / 2.268929 (2.706992)	2.279074 / 55.444624 (-53.165550)	1.952311 / 6.876477 (-4.924166)	2.167480 / 2.142072 (0.025408)	0.855882 / 4.805227 (-3.949346)	0.171378 / 6.500664 (-6.329287)	0.066731 / 0.075469 (-0.008738)

Benchmark: benchmark_map_filter.json

metric	filter	map fast-tokenizer batched	map identity	map identity batched	map no-op batched	map no-op batched numpy	map no-op batched pandas	map no-op batched pytorch	map no-op batched tensorflow
new / old (diff)	1.184226 / 1.841788 (-0.657561)	15.383396 / 8.074308 (7.309088)	15.069783 / 10.191392 (4.878391)	0.161489 / 0.680424 (-0.518935)	0.017763 / 0.534201 (-0.516438)	0.427103 / 0.579283 (-0.152180)	0.434295 / 0.434364 (-0.000069)	0.496848 / 0.540337 (-0.043489)	0.592572 / 1.386936 (-0.794364)

PyArrow==latest

Show updated benchmarks!

Benchmark: benchmark_array_xd.json

metric	read_batch_formatted_as_numpy after write_array2d	read_batch_formatted_as_numpy after write_flattened_sequence	read_batch_formatted_as_numpy after write_nested_sequence	read_batch_unformated after write_array2d	read_batch_unformated after write_flattened_sequence	read_batch_unformated after write_nested_sequence	read_col_formatted_as_numpy after write_array2d	read_col_formatted_as_numpy after write_flattened_sequence	read_col_formatted_as_numpy after write_nested_sequence	read_col_unformated after write_array2d	read_col_unformated after write_flattened_sequence	read_col_unformated after write_nested_sequence	read_formatted_as_numpy after write_array2d	read_formatted_as_numpy after write_flattened_sequence	read_formatted_as_numpy after write_nested_sequence	read_unformated after write_array2d	read_unformated after write_flattened_sequence	read_unformated after write_nested_sequence	write_array2d	write_flattened_sequence	write_nested_sequence
new / old (diff)	0.008014 / 0.011353 (-0.003339)	0.005607 / 0.011008 (-0.005401)	0.076826 / 0.038508 (0.038318)	0.035283 / 0.023109 (0.012174)	0.347809 / 0.275898 (0.071911)	0.382482 / 0.323480 (0.059003)	0.006276 / 0.007986 (-0.001709)	0.005978 / 0.004328 (0.001650)	0.074938 / 0.004250 (0.070687)	0.054323 / 0.037052 (0.017271)	0.344027 / 0.258489 (0.085538)	0.397623 / 0.293841 (0.103783)	0.037851 / 0.128546 (-0.090695)	0.012649 / 0.075646 (-0.062997)	0.086169 / 0.419271 (-0.333103)	0.051510 / 0.043533 (0.007977)	0.341112 / 0.255139 (0.085973)	0.357957 / 0.283200 (0.074757)	0.110949 / 0.141683 (-0.030734)	1.479573 / 1.452155 (0.027419)	1.578572 / 1.492716 (0.085855)

Benchmark: benchmark_getitem_100B.json

metric	get_batch_of_1024_random_rows	get_batch_of_1024_rows	get_first_row	get_last_row
new / old (diff)	0.310678 / 0.018006 (0.292672)	0.525504 / 0.000490 (0.525015)	0.000447 / 0.000200 (0.000247)	0.000060 / 0.000054 (0.000006)

Benchmark: benchmark_indices_mapping.json

metric	select	shard	shuffle	sort	train_test_split
new / old (diff)	0.031262 / 0.037411 (-0.006149)	0.113801 / 0.014526 (0.099275)	0.124967 / 0.176557 (-0.051590)	0.175226 / 0.737135 (-0.561909)	0.129377 / 0.296338 (-0.166962)

Benchmark: benchmark_iterating.json

metric	read 5000	read 50000	read_batch 50000 10	read_batch 50000 100	read_batch 50000 1000	read_formatted numpy 5000	read_formatted pandas 5000	read_formatted tensorflow 5000	read_formatted torch 5000	read_formatted_batch numpy 5000 10	read_formatted_batch numpy 5000 1000	shuffled read 5000	shuffled read 50000	shuffled read_batch 50000 10	shuffled read_batch 50000 100	shuffled read_batch 50000 1000	shuffled read_formatted numpy 5000	shuffled read_formatted_batch numpy 5000 10	shuffled read_formatted_batch numpy 5000 1000
new / old (diff)	0.420672 / 0.215209 (0.205463)	4.181337 / 2.077655 (2.103682)	1.985524 / 1.504120 (0.481404)	1.803468 / 1.541195 (0.262273)	1.952915 / 1.468490 (0.484425)	0.710928 / 4.584777 (-3.873849)	3.886245 / 3.745712 (0.140533)	3.737837 / 5.269862 (-1.532024)	1.806859 / 4.565676 (-2.758818)	0.088461 / 0.424275 (-0.335814)	0.013125 / 0.007607 (0.005518)	0.522410 / 0.226044 (0.296365)	5.232591 / 2.268929 (2.963663)	2.451188 / 55.444624 (-52.993437)	2.127725 / 6.876477 (-4.748751)	2.232859 / 2.142072 (0.090786)	0.854257 / 4.805227 (-3.950970)	0.171004 / 6.500664 (-6.329661)	0.066724 / 0.075469 (-0.008746)

Benchmark: benchmark_map_filter.json

metric	filter	map fast-tokenizer batched	map identity	map identity batched	map no-op batched	map no-op batched numpy	map no-op batched pandas	map no-op batched pytorch	map no-op batched tensorflow
new / old (diff)	1.257700 / 1.841788 (-0.584088)	15.738605 / 8.074308 (7.664297)	15.021698 / 10.191392 (4.830306)	0.147422 / 0.680424 (-0.533002)	0.017928 / 0.534201 (-0.516273)	0.428121 / 0.579283 (-0.151162)	0.432056 / 0.434364 (-0.002308)	0.498318 / 0.540337 (-0.042020)	0.591040 / 1.386936 (-0.795896)

[mariosasko]

Nice!

Eventually, it would be nice to merge GeneratorBasedBuilder and ArrowBasedBuilder to allow yielding Arrow tables in from_generator.

Then, we could do the following in to_iterable_dataset:

return IterableDataset.from_generator(
    Dataset._iter_shards, features=self.features, gen_kwargs={"shards": [shard.with_format("arrow") for shard in shards]}
)

[github-actions]

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

Benchmark: benchmark_array_xd.json

metric	read_batch_formatted_as_numpy after write_array2d	read_batch_formatted_as_numpy after write_flattened_sequence	read_batch_formatted_as_numpy after write_nested_sequence	read_batch_unformated after write_array2d	read_batch_unformated after write_flattened_sequence	read_batch_unformated after write_nested_sequence	read_col_formatted_as_numpy after write_array2d	read_col_formatted_as_numpy after write_flattened_sequence	read_col_formatted_as_numpy after write_nested_sequence	read_col_unformated after write_array2d	read_col_unformated after write_flattened_sequence	read_col_unformated after write_nested_sequence	read_formatted_as_numpy after write_array2d	read_formatted_as_numpy after write_flattened_sequence	read_formatted_as_numpy after write_nested_sequence	read_unformated after write_array2d	read_unformated after write_flattened_sequence	read_unformated after write_nested_sequence	write_array2d	write_flattened_sequence	write_nested_sequence
new / old (diff)	0.007014 / 0.011353 (-0.004339)	0.004792 / 0.011008 (-0.006216)	0.099822 / 0.038508 (0.061314)	0.029333 / 0.023109 (0.006224)	0.306453 / 0.275898 (0.030555)	0.344598 / 0.323480 (0.021118)	0.005121 / 0.007986 (-0.002865)	0.004850 / 0.004328 (0.000522)	0.076668 / 0.004250 (0.072417)	0.039980 / 0.037052 (0.002927)	0.312276 / 0.258489 (0.053787)	0.354722 / 0.293841 (0.060881)	0.031653 / 0.128546 (-0.096893)	0.011743 / 0.075646 (-0.063903)	0.322998 / 0.419271 (-0.096274)	0.042813 / 0.043533 (-0.000720)	0.308855 / 0.255139 (0.053716)	0.332650 / 0.283200 (0.049451)	0.087155 / 0.141683 (-0.054528)	1.454946 / 1.452155 (0.002791)	1.550589 / 1.492716 (0.057873)

Benchmark: benchmark_getitem_100B.json

metric	get_batch_of_1024_random_rows	get_batch_of_1024_rows	get_first_row	get_last_row
new / old (diff)	0.192921 / 0.018006 (0.174914)	0.411155 / 0.000490 (0.410666)	0.004779 / 0.000200 (0.004579)	0.000071 / 0.000054 (0.000017)

Benchmark: benchmark_indices_mapping.json

metric	select	shard	shuffle	sort	train_test_split
new / old (diff)	0.024462 / 0.037411 (-0.012950)	0.100320 / 0.014526 (0.085794)	0.105509 / 0.176557 (-0.071048)	0.168533 / 0.737135 (-0.568602)	0.110018 / 0.296338 (-0.186321)

Benchmark: benchmark_iterating.json

metric	read 5000	read 50000	read_batch 50000 10	read_batch 50000 100	read_batch 50000 1000	read_formatted numpy 5000	read_formatted pandas 5000	read_formatted tensorflow 5000	read_formatted torch 5000	read_formatted_batch numpy 5000 10	read_formatted_batch numpy 5000 1000	shuffled read 5000	shuffled read 50000	shuffled read_batch 50000 10	shuffled read_batch 50000 100	shuffled read_batch 50000 1000	shuffled read_formatted numpy 5000	shuffled read_formatted_batch numpy 5000 10	shuffled read_formatted_batch numpy 5000 1000
new / old (diff)	0.415025 / 0.215209 (0.199816)	4.144583 / 2.077655 (2.066928)	1.871627 / 1.504120 (0.367507)	1.671638 / 1.541195 (0.130443)	1.734458 / 1.468490 (0.265968)	0.693435 / 4.584777 (-3.891342)	3.487999 / 3.745712 (-0.257713)	3.196553 / 5.269862 (-2.073308)	1.628499 / 4.565676 (-2.937178)	0.082999 / 0.424275 (-0.341276)	0.012822 / 0.007607 (0.005215)	0.514904 / 0.226044 (0.288860)	5.157525 / 2.268929 (2.888596)	2.313093 / 55.444624 (-53.131531)	1.968335 / 6.876477 (-4.908142)	2.083462 / 2.142072 (-0.058610)	0.804485 / 4.805227 (-4.000742)	0.152290 / 6.500664 (-6.348374)	0.066813 / 0.075469 (-0.008656)

Benchmark: benchmark_map_filter.json

metric	filter	map fast-tokenizer batched	map identity	map identity batched	map no-op batched	map no-op batched numpy	map no-op batched pandas	map no-op batched pytorch	map no-op batched tensorflow
new / old (diff)	1.210370 / 1.841788 (-0.631418)	14.261779 / 8.074308 (6.187471)	14.268121 / 10.191392 (4.076729)	0.149216 / 0.680424 (-0.531207)	0.016529 / 0.534201 (-0.517672)	0.378814 / 0.579283 (-0.200469)	0.386304 / 0.434364 (-0.048060)	0.439653 / 0.540337 (-0.100684)	0.523658 / 1.386936 (-0.863278)

PyArrow==latest

Show updated benchmarks!

Benchmark: benchmark_array_xd.json

metric	read_batch_formatted_as_numpy after write_array2d	read_batch_formatted_as_numpy after write_flattened_sequence	read_batch_formatted_as_numpy after write_nested_sequence	read_batch_unformated after write_array2d	read_batch_unformated after write_flattened_sequence	read_batch_unformated after write_nested_sequence	read_col_formatted_as_numpy after write_array2d	read_col_formatted_as_numpy after write_flattened_sequence	read_col_formatted_as_numpy after write_nested_sequence	read_col_unformated after write_array2d	read_col_unformated after write_flattened_sequence	read_col_unformated after write_nested_sequence	read_formatted_as_numpy after write_array2d	read_formatted_as_numpy after write_flattened_sequence	read_formatted_as_numpy after write_nested_sequence	read_unformated after write_array2d	read_unformated after write_flattened_sequence	read_unformated after write_nested_sequence	write_array2d	write_flattened_sequence	write_nested_sequence
new / old (diff)	0.006979 / 0.011353 (-0.004374)	0.004718 / 0.011008 (-0.006290)	0.077023 / 0.038508 (0.038514)	0.029080 / 0.023109 (0.005971)	0.343145 / 0.275898 (0.067247)	0.380633 / 0.323480 (0.057153)	0.006057 / 0.007986 (-0.001928)	0.003541 / 0.004328 (-0.000788)	0.075773 / 0.004250 (0.071523)	0.039112 / 0.037052 (0.002060)	0.342355 / 0.258489 (0.083866)	0.386002 / 0.293841 (0.092161)	0.033238 / 0.128546 (-0.095308)	0.011696 / 0.075646 (-0.063950)	0.086178 / 0.419271 (-0.333093)	0.045219 / 0.043533 (0.001686)	0.360710 / 0.255139 (0.105571)	0.367490 / 0.283200 (0.084290)	0.093041 / 0.141683 (-0.048642)	1.523670 / 1.452155 (0.071516)	1.595280 / 1.492716 (0.102564)

Benchmark: benchmark_getitem_100B.json

metric	get_batch_of_1024_random_rows	get_batch_of_1024_rows	get_first_row	get_last_row
new / old (diff)	0.235888 / 0.018006 (0.217882)	0.410205 / 0.000490 (0.409715)	0.000405 / 0.000200 (0.000205)	0.000059 / 0.000054 (0.000005)

Benchmark: benchmark_indices_mapping.json

metric	select	shard	shuffle	sort	train_test_split
new / old (diff)	0.025752 / 0.037411 (-0.011659)	0.103343 / 0.014526 (0.088818)	0.108722 / 0.176557 (-0.067834)	0.159241 / 0.737135 (-0.577894)	0.113684 / 0.296338 (-0.182654)

Benchmark: benchmark_iterating.json

metric	read 5000	read 50000	read_batch 50000 10	read_batch 50000 100	read_batch 50000 1000	read_formatted numpy 5000	read_formatted pandas 5000	read_formatted tensorflow 5000	read_formatted torch 5000	read_formatted_batch numpy 5000 10	read_formatted_batch numpy 5000 1000	shuffled read 5000	shuffled read 50000	shuffled read_batch 50000 10	shuffled read_batch 50000 100	shuffled read_batch 50000 1000	shuffled read_formatted numpy 5000	shuffled read_formatted_batch numpy 5000 10	shuffled read_formatted_batch numpy 5000 1000
new / old (diff)	0.441809 / 0.215209 (0.226600)	4.410893 / 2.077655 (2.333238)	2.104061 / 1.504120 (0.599941)	1.854016 / 1.541195 (0.312821)	1.947100 / 1.468490 (0.478610)	0.697682 / 4.584777 (-3.887095)	3.467513 / 3.745712 (-0.278199)	1.911603 / 5.269862 (-3.358258)	1.187479 / 4.565676 (-3.378197)	0.083153 / 0.424275 (-0.341122)	0.012651 / 0.007607 (0.005044)	0.542081 / 0.226044 (0.316036)	5.444622 / 2.268929 (3.175693)	2.524236 / 55.444624 (-52.920388)	2.190463 / 6.876477 (-4.686014)	2.265764 / 2.142072 (0.123691)	0.810778 / 4.805227 (-3.994450)	0.152459 / 6.500664 (-6.348205)	0.067815 / 0.075469 (-0.007654)

Benchmark: benchmark_map_filter.json

metric	filter	map fast-tokenizer batched	map identity	map identity batched	map no-op batched	map no-op batched numpy	map no-op batched pandas	map no-op batched pytorch	map no-op batched tensorflow
new / old (diff)	1.334388 / 1.841788 (-0.507400)	14.640459 / 8.074308 (6.566151)	14.714874 / 10.191392 (4.523482)	0.153479 / 0.680424 (-0.526945)	0.016709 / 0.534201 (-0.517492)	0.379427 / 0.579283 (-0.199856)	0.391602 / 0.434364 (-0.042762)	0.438297 / 0.540337 (-0.102041)	0.524170 / 1.386936 (-0.862766)