Improve logging #6019

mariosasko · 2023-07-11T18:30:23Z

Adds the StreamHandler (as hfh and transformers do) to the library's logger to log INFO messages and logs the messages about "loading a cached result" (and some other warnings) as INFO

(Also removes the leave=False arg in the progress bars to be consistent with hfh and transformers - progress bars serve as an indicator that a result is not cached, so it makes more sense not to delete them)

Fix #2832, fix #1948, fix #5444

HuggingFaceDocBuilderDev · 2023-07-11T18:35:49Z

The documentation is not available anymore as the PR was closed or merged.

github-actions · 2023-07-11T18:37:54Z

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

Benchmark: benchmark_array_xd.json

metric	read_batch_formatted_as_numpy after write_array2d	read_batch_formatted_as_numpy after write_flattened_sequence	read_batch_formatted_as_numpy after write_nested_sequence	read_batch_unformated after write_array2d	read_batch_unformated after write_flattened_sequence	read_batch_unformated after write_nested_sequence	read_col_formatted_as_numpy after write_array2d	read_col_formatted_as_numpy after write_flattened_sequence	read_col_formatted_as_numpy after write_nested_sequence	read_col_unformated after write_array2d	read_col_unformated after write_flattened_sequence	read_col_unformated after write_nested_sequence	read_formatted_as_numpy after write_array2d	read_formatted_as_numpy after write_flattened_sequence	read_formatted_as_numpy after write_nested_sequence	read_unformated after write_array2d	read_unformated after write_flattened_sequence	read_unformated after write_nested_sequence	write_array2d	write_flattened_sequence	write_nested_sequence
new / old (diff)	0.007782 / 0.011353 (-0.003571)	0.004451 / 0.011008 (-0.006557)	0.099928 / 0.038508 (0.061420)	0.081534 / 0.023109 (0.058425)	0.379382 / 0.275898 (0.103484)	0.410652 / 0.323480 (0.087172)	0.005967 / 0.007986 (-0.002019)	0.003702 / 0.004328 (-0.000627)	0.076359 / 0.004250 (0.072109)	0.066721 / 0.037052 (0.029669)	0.383595 / 0.258489 (0.125106)	0.423854 / 0.293841 (0.130013)	0.032796 / 0.128546 (-0.095750)	0.009728 / 0.075646 (-0.065918)	0.344347 / 0.419271 (-0.074925)	0.056320 / 0.043533 (0.012788)	0.379974 / 0.255139 (0.124835)	0.401294 / 0.283200 (0.118094)	0.024110 / 0.141683 (-0.117572)	1.804194 / 1.452155 (0.352039)	1.860240 / 1.492716 (0.367523)

Benchmark: benchmark_getitem_100B.json

metric	get_batch_of_1024_random_rows	get_batch_of_1024_rows	get_first_row	get_last_row
new / old (diff)	0.233803 / 0.018006 (0.215797)	0.506893 / 0.000490 (0.506404)	0.003894 / 0.000200 (0.003694)	0.000090 / 0.000054 (0.000035)

Benchmark: benchmark_indices_mapping.json

metric	select	shard	shuffle	sort	train_test_split
new / old (diff)	0.033328 / 0.037411 (-0.004083)	0.098661 / 0.014526 (0.084136)	0.114971 / 0.176557 (-0.061586)	0.186815 / 0.737135 (-0.550321)	0.115490 / 0.296338 (-0.180848)

Benchmark: benchmark_iterating.json

metric	read 5000	read 50000	read_batch 50000 10	read_batch 50000 100	read_batch 50000 1000	read_formatted numpy 5000	read_formatted pandas 5000	read_formatted tensorflow 5000	read_formatted torch 5000	read_formatted_batch numpy 5000 10	read_formatted_batch numpy 5000 1000	shuffled read 5000	shuffled read 50000	shuffled read_batch 50000 10	shuffled read_batch 50000 100	shuffled read_batch 50000 1000	shuffled read_formatted numpy 5000	shuffled read_formatted_batch numpy 5000 10	shuffled read_formatted_batch numpy 5000 1000
new / old (diff)	0.422590 / 0.215209 (0.207381)	4.277189 / 2.077655 (2.199535)	2.095565 / 1.504120 (0.591445)	2.040825 / 1.541195 (0.499630)	2.162562 / 1.468490 (0.694072)	0.578602 / 4.584777 (-4.006175)	4.203474 / 3.745712 (0.457762)	6.674595 / 5.269862 (1.404734)	3.913251 / 4.565676 (-0.652426)	0.067777 / 0.424275 (-0.356498)	0.008716 / 0.007607 (0.001109)	0.548704 / 0.226044 (0.322660)	5.162120 / 2.268929 (2.893192)	2.600250 / 55.444624 (-52.844374)	2.232730 / 6.876477 (-4.643747)	2.485617 / 2.142072 (0.343544)	0.650872 / 4.805227 (-4.154355)	0.148022 / 6.500664 (-6.352642)	0.064795 / 0.075469 (-0.010674)

Benchmark: benchmark_map_filter.json

metric	filter	map fast-tokenizer batched	map identity	map identity batched	map no-op batched	map no-op batched numpy	map no-op batched pandas	map no-op batched pytorch	map no-op batched tensorflow
new / old (diff)	1.399439 / 1.841788 (-0.442349)	22.438959 / 8.074308 (14.364651)	16.447831 / 10.191392 (6.256439)	0.202003 / 0.680424 (-0.478421)	0.026200 / 0.534201 (-0.508001)	0.472966 / 0.579283 (-0.106317)	0.491621 / 0.434364 (0.057257)	0.551580 / 0.540337 (0.011242)	0.751420 / 1.386936 (-0.635516)

PyArrow==latest

Show updated benchmarks!

Benchmark: benchmark_array_xd.json

metric	read_batch_formatted_as_numpy after write_array2d	read_batch_formatted_as_numpy after write_flattened_sequence	read_batch_formatted_as_numpy after write_nested_sequence	read_batch_unformated after write_array2d	read_batch_unformated after write_flattened_sequence	read_batch_unformated after write_nested_sequence	read_col_formatted_as_numpy after write_array2d	read_col_formatted_as_numpy after write_flattened_sequence	read_col_formatted_as_numpy after write_nested_sequence	read_col_unformated after write_array2d	read_col_unformated after write_flattened_sequence	read_col_unformated after write_nested_sequence	read_formatted_as_numpy after write_array2d	read_formatted_as_numpy after write_flattened_sequence	read_formatted_as_numpy after write_nested_sequence	read_unformated after write_array2d	read_unformated after write_flattened_sequence	read_unformated after write_nested_sequence	write_array2d	write_flattened_sequence	write_nested_sequence
new / old (diff)	0.007241 / 0.011353 (-0.004112)	0.004434 / 0.011008 (-0.006574)	0.075872 / 0.038508 (0.037364)	0.080094 / 0.023109 (0.056985)	0.459244 / 0.275898 (0.183346)	0.492482 / 0.323480 (0.169002)	0.005791 / 0.007986 (-0.002194)	0.003657 / 0.004328 (-0.000671)	0.075214 / 0.004250 (0.070964)	0.064208 / 0.037052 (0.027156)	0.464195 / 0.258489 (0.205706)	0.497809 / 0.293841 (0.203968)	0.036301 / 0.128546 (-0.092245)	0.009855 / 0.075646 (-0.065791)	0.080826 / 0.419271 (-0.338445)	0.056700 / 0.043533 (0.013167)	0.452850 / 0.255139 (0.197711)	0.490738 / 0.283200 (0.207538)	0.024145 / 0.141683 (-0.117538)	1.689911 / 1.452155 (0.237757)	1.789803 / 1.492716 (0.297087)

Benchmark: benchmark_getitem_100B.json

metric	get_batch_of_1024_random_rows	get_batch_of_1024_rows	get_first_row	get_last_row
new / old (diff)	0.247741 / 0.018006 (0.229735)	0.486769 / 0.000490 (0.486279)	0.000418 / 0.000200 (0.000218)	0.000060 / 0.000054 (0.000005)

Benchmark: benchmark_indices_mapping.json

metric	select	shard	shuffle	sort	train_test_split
new / old (diff)	0.036317 / 0.037411 (-0.001094)	0.104943 / 0.014526 (0.090417)	0.120972 / 0.176557 (-0.055585)	0.188461 / 0.737135 (-0.548674)	0.120926 / 0.296338 (-0.175412)

Benchmark: benchmark_iterating.json

metric	read 5000	read 50000	read_batch 50000 10	read_batch 50000 100	read_batch 50000 1000	read_formatted numpy 5000	read_formatted pandas 5000	read_formatted tensorflow 5000	read_formatted torch 5000	read_formatted_batch numpy 5000 10	read_formatted_batch numpy 5000 1000	shuffled read 5000	shuffled read 50000	shuffled read_batch 50000 10	shuffled read_batch 50000 100	shuffled read_batch 50000 1000	shuffled read_formatted numpy 5000	shuffled read_formatted_batch numpy 5000 10	shuffled read_formatted_batch numpy 5000 1000
new / old (diff)	0.465788 / 0.215209 (0.250579)	4.662369 / 2.077655 (2.584714)	2.442241 / 1.504120 (0.938121)	2.266328 / 1.541195 (0.725133)	2.438998 / 1.468490 (0.970508)	0.531384 / 4.584777 (-4.053393)	4.125286 / 3.745712 (0.379574)	3.920912 / 5.269862 (-1.348950)	2.292149 / 4.565676 (-2.273528)	0.070146 / 0.424275 (-0.354129)	0.008887 / 0.007607 (0.001280)	0.598181 / 0.226044 (0.372137)	5.726454 / 2.268929 (3.457526)	3.081836 / 55.444624 (-52.362788)	2.683508 / 6.876477 (-4.192969)	2.587350 / 2.142072 (0.445278)	0.604736 / 4.805227 (-4.200491)	0.141303 / 6.500664 (-6.359362)	0.065020 / 0.075469 (-0.010449)

Benchmark: benchmark_map_filter.json

metric	filter	map fast-tokenizer batched	map identity	map identity batched	map no-op batched	map no-op batched numpy	map no-op batched pandas	map no-op batched pytorch	map no-op batched tensorflow
new / old (diff)	1.481850 / 1.841788 (-0.359938)	22.259592 / 8.074308 (14.185284)	16.304290 / 10.191392 (6.112898)	0.173514 / 0.680424 (-0.506909)	0.021590 / 0.534201 (-0.512611)	0.471753 / 0.579283 (-0.107531)	0.472132 / 0.434364 (0.037768)	0.563344 / 0.540337 (0.023007)	0.738509 / 1.386936 (-0.648427)

github-actions · 2023-07-11T18:38:43Z

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

Benchmark: benchmark_array_xd.json

metric	read_batch_formatted_as_numpy after write_array2d	read_batch_formatted_as_numpy after write_flattened_sequence	read_batch_formatted_as_numpy after write_nested_sequence	read_batch_unformated after write_array2d	read_batch_unformated after write_flattened_sequence	read_batch_unformated after write_nested_sequence	read_col_formatted_as_numpy after write_array2d	read_col_formatted_as_numpy after write_flattened_sequence	read_col_formatted_as_numpy after write_nested_sequence	read_col_unformated after write_array2d	read_col_unformated after write_flattened_sequence	read_col_unformated after write_nested_sequence	read_formatted_as_numpy after write_array2d	read_formatted_as_numpy after write_flattened_sequence	read_formatted_as_numpy after write_nested_sequence	read_unformated after write_array2d	read_unformated after write_flattened_sequence	read_unformated after write_nested_sequence	write_array2d	write_flattened_sequence	write_nested_sequence
new / old (diff)	0.005910 / 0.011353 (-0.005443)	0.004372 / 0.011008 (-0.006636)	0.081583 / 0.038508 (0.043075)	0.069598 / 0.023109 (0.046488)	0.346360 / 0.275898 (0.070462)	0.360733 / 0.323480 (0.037254)	0.004725 / 0.007986 (-0.003261)	0.003106 / 0.004328 (-0.001222)	0.059916 / 0.004250 (0.055666)	0.053242 / 0.037052 (0.016189)	0.353551 / 0.258489 (0.095062)	0.373052 / 0.293841 (0.079211)	0.029036 / 0.128546 (-0.099510)	0.007894 / 0.075646 (-0.067753)	0.284131 / 0.419271 (-0.135140)	0.049348 / 0.043533 (0.005815)	0.347409 / 0.255139 (0.092270)	0.355029 / 0.283200 (0.071830)	0.022511 / 0.141683 (-0.119171)	1.454495 / 1.452155 (0.002340)	1.439551 / 1.492716 (-0.053166)

Benchmark: benchmark_getitem_100B.json

metric	get_batch_of_1024_random_rows	get_batch_of_1024_rows	get_first_row	get_last_row
new / old (diff)	0.218889 / 0.018006 (0.200883)	0.478734 / 0.000490 (0.478244)	0.003758 / 0.000200 (0.003558)	0.000083 / 0.000054 (0.000029)

Benchmark: benchmark_indices_mapping.json

metric	select	shard	shuffle	sort	train_test_split
new / old (diff)	0.025759 / 0.037411 (-0.011653)	0.082511 / 0.014526 (0.067985)	0.087578 / 0.176557 (-0.088979)	0.137760 / 0.737135 (-0.599375)	0.093312 / 0.296338 (-0.203027)

Benchmark: benchmark_iterating.json

metric	read 5000	read 50000	read_batch 50000 10	read_batch 50000 100	read_batch 50000 1000	read_formatted numpy 5000	read_formatted pandas 5000	read_formatted tensorflow 5000	read_formatted torch 5000	read_formatted_batch numpy 5000 10	read_formatted_batch numpy 5000 1000	shuffled read 5000	shuffled read 50000	shuffled read_batch 50000 10	shuffled read_batch 50000 100	shuffled read_batch 50000 1000	shuffled read_formatted numpy 5000	shuffled read_formatted_batch numpy 5000 10	shuffled read_formatted_batch numpy 5000 1000
new / old (diff)	0.378963 / 0.215209 (0.163754)	3.645846 / 2.077655 (1.568191)	1.741135 / 1.504120 (0.237015)	1.599166 / 1.541195 (0.057972)	1.610817 / 1.468490 (0.142327)	0.459209 / 4.584777 (-4.125568)	3.484857 / 3.745712 (-0.260855)	3.928109 / 5.269862 (-1.341752)	2.419784 / 4.565676 (-2.145892)	0.051987 / 0.424275 (-0.372288)	0.006495 / 0.007607 (-0.001112)	0.427311 / 0.226044 (0.201267)	4.226378 / 2.268929 (1.957450)	2.212331 / 55.444624 (-53.232293)	1.916213 / 6.876477 (-4.960264)	1.978809 / 2.142072 (-0.163263)	0.547351 / 4.805227 (-4.257876)	0.121110 / 6.500664 (-6.379554)	0.054163 / 0.075469 (-0.021306)

Benchmark: benchmark_map_filter.json

metric	filter	map fast-tokenizer batched	map identity	map identity batched	map no-op batched	map no-op batched numpy	map no-op batched pandas	map no-op batched pytorch	map no-op batched tensorflow
new / old (diff)	1.228594 / 1.841788 (-0.613193)	19.410901 / 8.074308 (11.336593)	13.014722 / 10.191392 (2.823330)	0.156449 / 0.680424 (-0.523975)	0.021032 / 0.534201 (-0.513169)	0.403976 / 0.579283 (-0.175307)	0.413885 / 0.434364 (-0.020479)	0.470465 / 0.540337 (-0.069873)	0.641322 / 1.386936 (-0.745614)

PyArrow==latest

Show updated benchmarks!

Benchmark: benchmark_array_xd.json

metric	read_batch_formatted_as_numpy after write_array2d	read_batch_formatted_as_numpy after write_flattened_sequence	read_batch_formatted_as_numpy after write_nested_sequence	read_batch_unformated after write_array2d	read_batch_unformated after write_flattened_sequence	read_batch_unformated after write_nested_sequence	read_col_formatted_as_numpy after write_array2d	read_col_formatted_as_numpy after write_flattened_sequence	read_col_formatted_as_numpy after write_nested_sequence	read_col_unformated after write_array2d	read_col_unformated after write_flattened_sequence	read_col_unformated after write_nested_sequence	read_formatted_as_numpy after write_array2d	read_formatted_as_numpy after write_flattened_sequence	read_formatted_as_numpy after write_nested_sequence	read_unformated after write_array2d	read_unformated after write_flattened_sequence	read_unformated after write_nested_sequence	write_array2d	write_flattened_sequence	write_nested_sequence
new / old (diff)	0.007210 / 0.011353 (-0.004143)	0.003824 / 0.011008 (-0.007185)	0.058227 / 0.038508 (0.019719)	0.076211 / 0.023109 (0.053102)	0.336626 / 0.275898 (0.060728)	0.420542 / 0.323480 (0.097062)	0.006178 / 0.007986 (-0.001808)	0.003332 / 0.004328 (-0.000997)	0.058073 / 0.004250 (0.053823)	0.062485 / 0.037052 (0.025432)	0.386175 / 0.258489 (0.127686)	0.415659 / 0.293841 (0.121818)	0.031264 / 0.128546 (-0.097282)	0.007502 / 0.075646 (-0.068144)	0.072079 / 0.419271 (-0.347192)	0.055860 / 0.043533 (0.012327)	0.343508 / 0.255139 (0.088369)	0.437844 / 0.283200 (0.154645)	0.032852 / 0.141683 (-0.108831)	1.409241 / 1.452155 (-0.042913)	1.623949 / 1.492716 (0.131233)

Benchmark: benchmark_getitem_100B.json

metric	get_batch_of_1024_random_rows	get_batch_of_1024_rows	get_first_row	get_last_row
new / old (diff)	0.207511 / 0.018006 (0.189504)	0.464149 / 0.000490 (0.463660)	0.003248 / 0.000200 (0.003048)	0.000226 / 0.000054 (0.000172)

Benchmark: benchmark_indices_mapping.json

metric	select	shard	shuffle	sort	train_test_split
new / old (diff)	0.030767 / 0.037411 (-0.006645)	0.079169 / 0.014526 (0.064643)	0.093111 / 0.176557 (-0.083445)	0.153369 / 0.737135 (-0.583767)	0.092939 / 0.296338 (-0.203400)

Benchmark: benchmark_iterating.json

metric	read 5000	read 50000	read_batch 50000 10	read_batch 50000 100	read_batch 50000 1000	read_formatted numpy 5000	read_formatted pandas 5000	read_formatted tensorflow 5000	read_formatted torch 5000	read_formatted_batch numpy 5000 10	read_formatted_batch numpy 5000 1000	shuffled read 5000	shuffled read 50000	shuffled read_batch 50000 10	shuffled read_batch 50000 100	shuffled read_batch 50000 1000	shuffled read_formatted numpy 5000	shuffled read_formatted_batch numpy 5000 10	shuffled read_formatted_batch numpy 5000 1000
new / old (diff)	0.375602 / 0.215209 (0.160392)	3.968612 / 2.077655 (1.890957)	2.081749 / 1.504120 (0.577629)	1.899772 / 1.541195 (0.358577)	1.847923 / 1.468490 (0.379433)	0.442867 / 4.584777 (-4.141910)	3.646664 / 3.745712 (-0.099048)	5.870600 / 5.269862 (0.600739)	3.356698 / 4.565676 (-1.208979)	0.051422 / 0.424275 (-0.372853)	0.006006 / 0.007607 (-0.001601)	0.442439 / 0.226044 (0.216395)	4.466256 / 2.268929 (2.197328)	2.483832 / 55.444624 (-52.960792)	2.105612 / 6.876477 (-4.770865)	2.060650 / 2.142072 (-0.081422)	0.531119 / 4.805227 (-4.274108)	0.123436 / 6.500664 (-6.377228)	0.059838 / 0.075469 (-0.015632)

Benchmark: benchmark_map_filter.json

metric	filter	map fast-tokenizer batched	map identity	map identity batched	map no-op batched	map no-op batched numpy	map no-op batched pandas	map no-op batched pytorch	map no-op batched tensorflow
new / old (diff)	1.283042 / 1.841788 (-0.558746)	19.688251 / 8.074308 (11.613943)	13.346386 / 10.191392 (3.154994)	0.197463 / 0.680424 (-0.482961)	0.018484 / 0.534201 (-0.515717)	0.391727 / 0.579283 (-0.187556)	0.425061 / 0.434364 (-0.009303)	0.448177 / 0.540337 (-0.092160)	0.653694 / 1.386936 (-0.733242)

github-actions · 2023-07-11T18:49:11Z

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

Benchmark: benchmark_array_xd.json

metric	read_batch_formatted_as_numpy after write_array2d	read_batch_formatted_as_numpy after write_flattened_sequence	read_batch_formatted_as_numpy after write_nested_sequence	read_batch_unformated after write_array2d	read_batch_unformated after write_flattened_sequence	read_batch_unformated after write_nested_sequence	read_col_formatted_as_numpy after write_array2d	read_col_formatted_as_numpy after write_flattened_sequence	read_col_formatted_as_numpy after write_nested_sequence	read_col_unformated after write_array2d	read_col_unformated after write_flattened_sequence	read_col_unformated after write_nested_sequence	read_formatted_as_numpy after write_array2d	read_formatted_as_numpy after write_flattened_sequence	read_formatted_as_numpy after write_nested_sequence	read_unformated after write_array2d	read_unformated after write_flattened_sequence	read_unformated after write_nested_sequence	write_array2d	write_flattened_sequence	write_nested_sequence
new / old (diff)	0.008966 / 0.011353 (-0.002387)	0.005195 / 0.011008 (-0.005813)	0.102879 / 0.038508 (0.064371)	0.090902 / 0.023109 (0.067792)	0.434397 / 0.275898 (0.158498)	0.454013 / 0.323480 (0.130534)	0.008507 / 0.007986 (0.000521)	0.005000 / 0.004328 (0.000671)	0.075789 / 0.004250 (0.071538)	0.067608 / 0.037052 (0.030555)	0.435091 / 0.258489 (0.176602)	0.469411 / 0.293841 (0.175570)	0.050859 / 0.128546 (-0.077687)	0.013560 / 0.075646 (-0.062086)	0.345473 / 0.419271 (-0.073799)	0.094974 / 0.043533 (0.051441)	0.429626 / 0.255139 (0.174487)	0.434290 / 0.283200 (0.151090)	0.052269 / 0.141683 (-0.089413)	1.700549 / 1.452155 (0.248395)	1.890693 / 1.492716 (0.397976)

Benchmark: benchmark_getitem_100B.json

metric	get_batch_of_1024_random_rows	get_batch_of_1024_rows	get_first_row	get_last_row
new / old (diff)	0.296618 / 0.018006 (0.278612)	0.613908 / 0.000490 (0.613419)	0.000484 / 0.000200 (0.000284)	0.000086 / 0.000054 (0.000032)

Benchmark: benchmark_indices_mapping.json

metric	select	shard	shuffle	sort	train_test_split
new / old (diff)	0.034346 / 0.037411 (-0.003065)	0.096836 / 0.014526 (0.082310)	0.113332 / 0.176557 (-0.063224)	0.194464 / 0.737135 (-0.542671)	0.111732 / 0.296338 (-0.184606)

Benchmark: benchmark_iterating.json

metric	read 5000	read 50000	read_batch 50000 10	read_batch 50000 100	read_batch 50000 1000	read_formatted numpy 5000	read_formatted pandas 5000	read_formatted tensorflow 5000	read_formatted torch 5000	read_formatted_batch numpy 5000 10	read_formatted_batch numpy 5000 1000	shuffled read 5000	shuffled read 50000	shuffled read_batch 50000 10	shuffled read_batch 50000 100	shuffled read_batch 50000 1000	shuffled read_formatted numpy 5000	shuffled read_formatted_batch numpy 5000 10	shuffled read_formatted_batch numpy 5000 1000
new / old (diff)	0.624954 / 0.215209 (0.409745)	6.442193 / 2.077655 (4.364538)	2.818331 / 1.504120 (1.314211)	2.529607 / 1.541195 (0.988413)	2.549026 / 1.468490 (1.080536)	0.967367 / 4.584777 (-3.617410)	5.446885 / 3.745712 (1.701173)	6.259099 / 5.269862 (0.989237)	3.652936 / 4.565676 (-0.912740)	0.106420 / 0.424275 (-0.317855)	0.011293 / 0.007607 (0.003686)	0.772026 / 0.226044 (0.545982)	7.823986 / 2.268929 (5.555057)	3.725328 / 55.444624 (-51.719297)	2.851489 / 6.876477 (-4.024988)	3.013722 / 2.142072 (0.871649)	1.045090 / 4.805227 (-3.760137)	0.213174 / 6.500664 (-6.287490)	0.077104 / 0.075469 (0.001635)

Benchmark: benchmark_map_filter.json

metric	filter	map fast-tokenizer batched	map identity	map identity batched	map no-op batched	map no-op batched numpy	map no-op batched pandas	map no-op batched pytorch	map no-op batched tensorflow
new / old (diff)	1.657135 / 1.841788 (-0.184652)	24.547604 / 8.074308 (16.473296)	19.989533 / 10.191392 (9.798141)	0.257139 / 0.680424 (-0.423285)	0.028448 / 0.534201 (-0.505753)	0.490801 / 0.579283 (-0.088482)	0.628072 / 0.434364 (0.193708)	0.584873 / 0.540337 (0.044536)	0.825258 / 1.386936 (-0.561678)

PyArrow==latest

Show updated benchmarks!

Benchmark: benchmark_array_xd.json

metric	read_batch_formatted_as_numpy after write_array2d	read_batch_formatted_as_numpy after write_flattened_sequence	read_batch_formatted_as_numpy after write_nested_sequence	read_batch_unformated after write_array2d	read_batch_unformated after write_flattened_sequence	read_batch_unformated after write_nested_sequence	read_col_formatted_as_numpy after write_array2d	read_col_formatted_as_numpy after write_flattened_sequence	read_col_formatted_as_numpy after write_nested_sequence	read_col_unformated after write_array2d	read_col_unformated after write_flattened_sequence	read_col_unformated after write_nested_sequence	read_formatted_as_numpy after write_array2d	read_formatted_as_numpy after write_flattened_sequence	read_formatted_as_numpy after write_nested_sequence	read_unformated after write_array2d	read_unformated after write_flattened_sequence	read_unformated after write_nested_sequence	write_array2d	write_flattened_sequence	write_nested_sequence
new / old (diff)	0.009258 / 0.011353 (-0.002095)	0.005660 / 0.011008 (-0.005348)	0.080577 / 0.038508 (0.042069)	0.095786 / 0.023109 (0.072676)	0.473334 / 0.275898 (0.197436)	0.527962 / 0.323480 (0.204482)	0.006537 / 0.007986 (-0.001449)	0.004411 / 0.004328 (0.000083)	0.080702 / 0.004250 (0.076452)	0.077020 / 0.037052 (0.039968)	0.483205 / 0.258489 (0.224716)	0.556916 / 0.293841 (0.263076)	0.047670 / 0.128546 (-0.080877)	0.016647 / 0.075646 (-0.058999)	0.090653 / 0.419271 (-0.328619)	0.062122 / 0.043533 (0.018589)	0.498326 / 0.255139 (0.243187)	0.546572 / 0.283200 (0.263372)	0.037525 / 0.141683 (-0.104157)	1.869520 / 1.452155 (0.417365)	1.915335 / 1.492716 (0.422619)

Benchmark: benchmark_getitem_100B.json

metric	get_batch_of_1024_random_rows	get_batch_of_1024_rows	get_first_row	get_last_row
new / old (diff)	0.248287 / 0.018006 (0.230281)	0.611440 / 0.000490 (0.610950)	0.004102 / 0.000200 (0.003902)	0.000132 / 0.000054 (0.000078)

Benchmark: benchmark_indices_mapping.json

metric	select	shard	shuffle	sort	train_test_split
new / old (diff)	0.038228 / 0.037411 (0.000817)	0.103510 / 0.014526 (0.088984)	0.114337 / 0.176557 (-0.062219)	0.189662 / 0.737135 (-0.547473)	0.119078 / 0.296338 (-0.177260)

Benchmark: benchmark_iterating.json

metric	read 5000	read 50000	read_batch 50000 10	read_batch 50000 100	read_batch 50000 1000	read_formatted numpy 5000	read_formatted pandas 5000	read_formatted tensorflow 5000	read_formatted torch 5000	read_formatted_batch numpy 5000 10	read_formatted_batch numpy 5000 1000	shuffled read 5000	shuffled read 50000	shuffled read_batch 50000 10	shuffled read_batch 50000 100	shuffled read_batch 50000 1000	shuffled read_formatted numpy 5000	shuffled read_formatted_batch numpy 5000 10	shuffled read_formatted_batch numpy 5000 1000
new / old (diff)	0.606622 / 0.215209 (0.391413)	6.053900 / 2.077655 (3.976246)	2.857972 / 1.504120 (1.353852)	2.549756 / 1.541195 (1.008561)	2.584557 / 1.468490 (1.116067)	0.930431 / 4.584777 (-3.654346)	5.524077 / 3.745712 (1.778365)	7.858406 / 5.269862 (2.588545)	4.890697 / 4.565676 (0.325020)	0.095356 / 0.424275 (-0.328919)	0.008614 / 0.007607 (0.001007)	0.774227 / 0.226044 (0.548182)	7.470215 / 2.268929 (5.201287)	3.784820 / 55.444624 (-51.659805)	3.199364 / 6.876477 (-3.677113)	3.212002 / 2.142072 (1.069929)	1.054104 / 4.805227 (-3.751123)	0.226044 / 6.500664 (-6.274620)	0.092237 / 0.075469 (0.016768)

Benchmark: benchmark_map_filter.json

metric	filter	map fast-tokenizer batched	map identity	map identity batched	map no-op batched	map no-op batched numpy	map no-op batched pandas	map no-op batched pytorch	map no-op batched tensorflow
new / old (diff)	1.801054 / 1.841788 (-0.040734)	24.220404 / 8.074308 (16.146096)	21.652936 / 10.191392 (11.461544)	0.247004 / 0.680424 (-0.433420)	0.029651 / 0.534201 (-0.504550)	0.475702 / 0.579283 (-0.103581)	0.621121 / 0.434364 (0.186757)	0.570489 / 0.540337 (0.030151)	0.768840 / 1.386936 (-0.618096)

github-actions · 2023-07-11T19:37:25Z

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

Benchmark: benchmark_array_xd.json

metric	read_batch_formatted_as_numpy after write_array2d	read_batch_formatted_as_numpy after write_flattened_sequence	read_batch_formatted_as_numpy after write_nested_sequence	read_batch_unformated after write_array2d	read_batch_unformated after write_flattened_sequence	read_batch_unformated after write_nested_sequence	read_col_formatted_as_numpy after write_array2d	read_col_formatted_as_numpy after write_flattened_sequence	read_col_formatted_as_numpy after write_nested_sequence	read_col_unformated after write_array2d	read_col_unformated after write_flattened_sequence	read_col_unformated after write_nested_sequence	read_formatted_as_numpy after write_array2d	read_formatted_as_numpy after write_flattened_sequence	read_formatted_as_numpy after write_nested_sequence	read_unformated after write_array2d	read_unformated after write_flattened_sequence	read_unformated after write_nested_sequence	write_array2d	write_flattened_sequence	write_nested_sequence
new / old (diff)	0.009223 / 0.011353 (-0.002130)	0.005750 / 0.011008 (-0.005258)	0.105264 / 0.038508 (0.066756)	0.088478 / 0.023109 (0.065369)	0.461119 / 0.275898 (0.185221)	0.481115 / 0.323480 (0.157636)	0.006366 / 0.007986 (-0.001619)	0.004515 / 0.004328 (0.000186)	0.079296 / 0.004250 (0.075045)	0.063483 / 0.037052 (0.026430)	0.444490 / 0.258489 (0.186001)	0.496474 / 0.293841 (0.202634)	0.048568 / 0.128546 (-0.079978)	0.013574 / 0.075646 (-0.062073)	0.379213 / 0.419271 (-0.040059)	0.086464 / 0.043533 (0.042932)	0.437526 / 0.255139 (0.182387)	0.447117 / 0.283200 (0.163917)	0.049502 / 0.141683 (-0.092180)	1.749146 / 1.452155 (0.296992)	1.831082 / 1.492716 (0.338365)

Benchmark: benchmark_getitem_100B.json

metric	get_batch_of_1024_random_rows	get_batch_of_1024_rows	get_first_row	get_last_row
new / old (diff)	0.268205 / 0.018006 (0.250199)	0.627406 / 0.000490 (0.626917)	0.005439 / 0.000200 (0.005239)	0.000128 / 0.000054 (0.000074)

Benchmark: benchmark_indices_mapping.json

metric	select	shard	shuffle	sort	train_test_split
new / old (diff)	0.030564 / 0.037411 (-0.006848)	0.096365 / 0.014526 (0.081840)	0.117484 / 0.176557 (-0.059072)	0.189104 / 0.737135 (-0.548032)	0.118073 / 0.296338 (-0.178266)

Benchmark: benchmark_iterating.json

metric	read 5000	read 50000	read_batch 50000 10	read_batch 50000 100	read_batch 50000 1000	read_formatted numpy 5000	read_formatted pandas 5000	read_formatted tensorflow 5000	read_formatted torch 5000	read_formatted_batch numpy 5000 10	read_formatted_batch numpy 5000 1000	shuffled read 5000	shuffled read 50000	shuffled read_batch 50000 10	shuffled read_batch 50000 100	shuffled read_batch 50000 1000	shuffled read_formatted numpy 5000	shuffled read_formatted_batch numpy 5000 10	shuffled read_formatted_batch numpy 5000 1000
new / old (diff)	0.618229 / 0.215209 (0.403019)	6.437853 / 2.077655 (4.360199)	2.789946 / 1.504120 (1.285826)	2.339245 / 1.541195 (0.798050)	2.588779 / 1.468490 (1.120289)	0.921008 / 4.584777 (-3.663769)	5.402940 / 3.745712 (1.657227)	4.818783 / 5.269862 (-0.451078)	3.162259 / 4.565676 (-1.403417)	0.108501 / 0.424275 (-0.315774)	0.009384 / 0.007607 (0.001777)	0.766811 / 0.226044 (0.540766)	7.624629 / 2.268929 (5.355701)	3.442420 / 55.444624 (-52.002204)	2.759967 / 6.876477 (-4.116510)	3.049644 / 2.142072 (0.907572)	1.113308 / 4.805227 (-3.691919)	0.223923 / 6.500664 (-6.276741)	0.079156 / 0.075469 (0.003687)

Benchmark: benchmark_map_filter.json

metric	filter	map fast-tokenizer batched	map identity	map identity batched	map no-op batched	map no-op batched numpy	map no-op batched pandas	map no-op batched pytorch	map no-op batched tensorflow
new / old (diff)	1.683318 / 1.841788 (-0.158470)	25.062141 / 8.074308 (16.987833)	21.777131 / 10.191392 (11.585739)	0.266939 / 0.680424 (-0.413485)	0.029670 / 0.534201 (-0.504531)	0.476761 / 0.579283 (-0.102522)	0.622080 / 0.434364 (0.187716)	0.601781 / 0.540337 (0.061443)	0.785126 / 1.386936 (-0.601811)

PyArrow==latest

Show updated benchmarks!

Benchmark: benchmark_array_xd.json

metric	read_batch_formatted_as_numpy after write_array2d	read_batch_formatted_as_numpy after write_flattened_sequence	read_batch_formatted_as_numpy after write_nested_sequence	read_batch_unformated after write_array2d	read_batch_unformated after write_flattened_sequence	read_batch_unformated after write_nested_sequence	read_col_formatted_as_numpy after write_array2d	read_col_formatted_as_numpy after write_flattened_sequence	read_col_formatted_as_numpy after write_nested_sequence	read_col_unformated after write_array2d	read_col_unformated after write_flattened_sequence	read_col_unformated after write_nested_sequence	read_formatted_as_numpy after write_array2d	read_formatted_as_numpy after write_flattened_sequence	read_formatted_as_numpy after write_nested_sequence	read_unformated after write_array2d	read_unformated after write_flattened_sequence	read_unformated after write_nested_sequence	write_array2d	write_flattened_sequence	write_nested_sequence
new / old (diff)	0.010198 / 0.011353 (-0.001155)	0.005777 / 0.011008 (-0.005231)	0.083003 / 0.038508 (0.044495)	0.093411 / 0.023109 (0.070302)	0.496178 / 0.275898 (0.220280)	0.554670 / 0.323480 (0.231190)	0.008351 / 0.007986 (0.000365)	0.004678 / 0.004328 (0.000350)	0.083631 / 0.004250 (0.079381)	0.075538 / 0.037052 (0.038485)	0.492410 / 0.258489 (0.233921)	0.545209 / 0.293841 (0.251368)	0.048365 / 0.128546 (-0.080181)	0.014219 / 0.075646 (-0.061427)	0.100749 / 0.419271 (-0.318523)	0.063431 / 0.043533 (0.019898)	0.511115 / 0.255139 (0.255976)	0.532965 / 0.283200 (0.249765)	0.037968 / 0.141683 (-0.103715)	1.940268 / 1.452155 (0.488113)	2.032934 / 1.492716 (0.540217)

Benchmark: benchmark_getitem_100B.json

metric	get_batch_of_1024_random_rows	get_batch_of_1024_rows	get_first_row	get_last_row
new / old (diff)	0.238179 / 0.018006 (0.220172)	0.605767 / 0.000490 (0.605277)	0.004033 / 0.000200 (0.003833)	0.000125 / 0.000054 (0.000071)

Benchmark: benchmark_indices_mapping.json

metric	select	shard	shuffle	sort	train_test_split
new / old (diff)	0.036436 / 0.037411 (-0.000975)	0.108034 / 0.014526 (0.093509)	0.118624 / 0.176557 (-0.057933)	0.183079 / 0.737135 (-0.554056)	0.121739 / 0.296338 (-0.174600)

Benchmark: benchmark_iterating.json

metric	read 5000	read 50000	read_batch 50000 10	read_batch 50000 100	read_batch 50000 1000	read_formatted numpy 5000	read_formatted pandas 5000	read_formatted tensorflow 5000	read_formatted torch 5000	read_formatted_batch numpy 5000 10	read_formatted_batch numpy 5000 1000	shuffled read 5000	shuffled read 50000	shuffled read_batch 50000 10	shuffled read_batch 50000 100	shuffled read_batch 50000 1000	shuffled read_formatted numpy 5000	shuffled read_formatted_batch numpy 5000 10	shuffled read_formatted_batch numpy 5000 1000
new / old (diff)	0.630538 / 0.215209 (0.415329)	6.552184 / 2.077655 (4.474529)	3.003412 / 1.504120 (1.499292)	2.669026 / 1.541195 (1.127832)	2.791109 / 1.468490 (1.322619)	0.884003 / 4.584777 (-3.700774)	5.538660 / 3.745712 (1.792947)	5.126708 / 5.269862 (-0.143154)	3.120825 / 4.565676 (-1.444852)	0.101178 / 0.424275 (-0.323097)	0.009027 / 0.007607 (0.001420)	0.785914 / 0.226044 (0.559869)	7.994720 / 2.268929 (5.725792)	4.061996 / 55.444624 (-51.382629)	3.263230 / 6.876477 (-3.613247)	3.288622 / 2.142072 (1.146550)	1.141867 / 4.805227 (-3.663360)	0.255287 / 6.500664 (-6.245378)	0.100637 / 0.075469 (0.025168)

Benchmark: benchmark_map_filter.json

metric	filter	map fast-tokenizer batched	map identity	map identity batched	map no-op batched	map no-op batched numpy	map no-op batched pandas	map no-op batched pytorch	map no-op batched tensorflow
new / old (diff)	1.769821 / 1.841788 (-0.071967)	24.994008 / 8.074308 (16.919700)	21.765971 / 10.191392 (11.574579)	0.268493 / 0.680424 (-0.411931)	0.028047 / 0.534201 (-0.506154)	0.489472 / 0.579283 (-0.089811)	0.594809 / 0.434364 (0.160445)	0.613578 / 0.540337 (0.073241)	0.879360 / 1.386936 (-0.507576)

github-actions · 2023-07-11T20:10:41Z

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

Benchmark: benchmark_array_xd.json

metric	read_batch_formatted_as_numpy after write_array2d	read_batch_formatted_as_numpy after write_flattened_sequence	read_batch_formatted_as_numpy after write_nested_sequence	read_batch_unformated after write_array2d	read_batch_unformated after write_flattened_sequence	read_batch_unformated after write_nested_sequence	read_col_formatted_as_numpy after write_array2d	read_col_formatted_as_numpy after write_flattened_sequence	read_col_formatted_as_numpy after write_nested_sequence	read_col_unformated after write_array2d	read_col_unformated after write_flattened_sequence	read_col_unformated after write_nested_sequence	read_formatted_as_numpy after write_array2d	read_formatted_as_numpy after write_flattened_sequence	read_formatted_as_numpy after write_nested_sequence	read_unformated after write_array2d	read_unformated after write_flattened_sequence	read_unformated after write_nested_sequence	write_array2d	write_flattened_sequence	write_nested_sequence
new / old (diff)	0.006003 / 0.011353 (-0.005350)	0.003590 / 0.011008 (-0.007418)	0.084657 / 0.038508 (0.046149)	0.057884 / 0.023109 (0.034775)	0.318347 / 0.275898 (0.042449)	0.345976 / 0.323480 (0.022496)	0.004706 / 0.007986 (-0.003279)	0.002921 / 0.004328 (-0.001407)	0.061850 / 0.004250 (0.057600)	0.050558 / 0.037052 (0.013505)	0.320877 / 0.258489 (0.062388)	0.356062 / 0.293841 (0.062222)	0.027511 / 0.128546 (-0.101035)	0.007954 / 0.075646 (-0.067693)	0.260290 / 0.419271 (-0.158981)	0.051207 / 0.043533 (0.007674)	0.334423 / 0.255139 (0.079284)	0.338575 / 0.283200 (0.055375)	0.022330 / 0.141683 (-0.119353)	1.445446 / 1.452155 (-0.006709)	1.500626 / 1.492716 (0.007910)

Benchmark: benchmark_getitem_100B.json

metric	get_batch_of_1024_random_rows	get_batch_of_1024_rows	get_first_row	get_last_row
new / old (diff)	0.192440 / 0.018006 (0.174433)	0.428455 / 0.000490 (0.427965)	0.000318 / 0.000200 (0.000118)	0.000056 / 0.000054 (0.000002)

Benchmark: benchmark_indices_mapping.json

metric	select	shard	shuffle	sort	train_test_split
new / old (diff)	0.022933 / 0.037411 (-0.014478)	0.072795 / 0.014526 (0.058269)	0.081149 / 0.176557 (-0.095407)	0.142941 / 0.737135 (-0.594195)	0.082410 / 0.296338 (-0.213928)

Benchmark: benchmark_iterating.json

metric	read 5000	read 50000	read_batch 50000 10	read_batch 50000 100	read_batch 50000 1000	read_formatted numpy 5000	read_formatted pandas 5000	read_formatted tensorflow 5000	read_formatted torch 5000	read_formatted_batch numpy 5000 10	read_formatted_batch numpy 5000 1000	shuffled read 5000	shuffled read 50000	shuffled read_batch 50000 10	shuffled read_batch 50000 100	shuffled read_batch 50000 1000	shuffled read_formatted numpy 5000	shuffled read_formatted_batch numpy 5000 10	shuffled read_formatted_batch numpy 5000 1000
new / old (diff)	0.405220 / 0.215209 (0.190011)	4.048585 / 2.077655 (1.970931)	2.027908 / 1.504120 (0.523788)	1.887828 / 1.541195 (0.346633)	2.131780 / 1.468490 (0.663290)	0.502847 / 4.584777 (-4.081930)	3.069498 / 3.745712 (-0.676215)	4.094774 / 5.269862 (-1.175088)	2.544004 / 4.565676 (-2.021673)	0.059540 / 0.424275 (-0.364735)	0.006501 / 0.007607 (-0.001106)	0.477218 / 0.226044 (0.251173)	4.764961 / 2.268929 (2.496032)	2.434594 / 55.444624 (-53.010030)	2.104833 / 6.876477 (-4.771644)	2.263059 / 2.142072 (0.120987)	0.591755 / 4.805227 (-4.213472)	0.131167 / 6.500664 (-6.369497)	0.061808 / 0.075469 (-0.013661)

Benchmark: benchmark_map_filter.json

metric	filter	map fast-tokenizer batched	map identity	map identity batched	map no-op batched	map no-op batched numpy	map no-op batched pandas	map no-op batched pytorch	map no-op batched tensorflow
new / old (diff)	1.345364 / 1.841788 (-0.496424)	18.122584 / 8.074308 (10.048276)	13.318689 / 10.191392 (3.127297)	0.144526 / 0.680424 (-0.535898)	0.016997 / 0.534201 (-0.517204)	0.336036 / 0.579283 (-0.243247)	0.359532 / 0.434364 (-0.074832)	0.386945 / 0.540337 (-0.153392)	0.538659 / 1.386936 (-0.848277)

PyArrow==latest

Show updated benchmarks!

Benchmark: benchmark_array_xd.json

metric	read_batch_formatted_as_numpy after write_array2d	read_batch_formatted_as_numpy after write_flattened_sequence	read_batch_formatted_as_numpy after write_nested_sequence	read_batch_unformated after write_array2d	read_batch_unformated after write_flattened_sequence	read_batch_unformated after write_nested_sequence	read_col_formatted_as_numpy after write_array2d	read_col_formatted_as_numpy after write_flattened_sequence	read_col_formatted_as_numpy after write_nested_sequence	read_col_unformated after write_array2d	read_col_unformated after write_flattened_sequence	read_col_unformated after write_nested_sequence	read_formatted_as_numpy after write_array2d	read_formatted_as_numpy after write_flattened_sequence	read_formatted_as_numpy after write_nested_sequence	read_unformated after write_array2d	read_unformated after write_flattened_sequence	read_unformated after write_nested_sequence	write_array2d	write_flattened_sequence	write_nested_sequence
new / old (diff)	0.006088 / 0.011353 (-0.005265)	0.003684 / 0.011008 (-0.007324)	0.062340 / 0.038508 (0.023832)	0.058461 / 0.023109 (0.035352)	0.360134 / 0.275898 (0.084236)	0.393298 / 0.323480 (0.069818)	0.004664 / 0.007986 (-0.003322)	0.002909 / 0.004328 (-0.001420)	0.062668 / 0.004250 (0.058418)	0.050145 / 0.037052 (0.013092)	0.361897 / 0.258489 (0.103408)	0.402008 / 0.293841 (0.108167)	0.027491 / 0.128546 (-0.101055)	0.008113 / 0.075646 (-0.067534)	0.068114 / 0.419271 (-0.351157)	0.043303 / 0.043533 (-0.000230)	0.360569 / 0.255139 (0.105430)	0.387144 / 0.283200 (0.103944)	0.020194 / 0.141683 (-0.121489)	1.418066 / 1.452155 (-0.034089)	1.475640 / 1.492716 (-0.017076)

Benchmark: benchmark_getitem_100B.json

metric	get_batch_of_1024_random_rows	get_batch_of_1024_rows	get_first_row	get_last_row
new / old (diff)	0.200291 / 0.018006 (0.182285)	0.432298 / 0.000490 (0.431809)	0.003303 / 0.000200 (0.003103)	0.000075 / 0.000054 (0.000020)

Benchmark: benchmark_indices_mapping.json

metric	select	shard	shuffle	sort	train_test_split
new / old (diff)	0.027749 / 0.037411 (-0.009662)	0.081890 / 0.014526 (0.067364)	0.094319 / 0.176557 (-0.082238)	0.148646 / 0.737135 (-0.588490)	0.091830 / 0.296338 (-0.204509)

Benchmark: benchmark_iterating.json

metric	read 5000	read 50000	read_batch 50000 10	read_batch 50000 100	read_batch 50000 1000	read_formatted numpy 5000	read_formatted pandas 5000	read_formatted tensorflow 5000	read_formatted torch 5000	read_formatted_batch numpy 5000 10	read_formatted_batch numpy 5000 1000	shuffled read 5000	shuffled read 50000	shuffled read_batch 50000 10	shuffled read_batch 50000 100	shuffled read_batch 50000 1000	shuffled read_formatted numpy 5000	shuffled read_formatted_batch numpy 5000 10	shuffled read_formatted_batch numpy 5000 1000
new / old (diff)	0.433546 / 0.215209 (0.218337)	4.326855 / 2.077655 (2.249200)	2.230186 / 1.504120 (0.726066)	2.052524 / 1.541195 (0.511329)	2.117270 / 1.468490 (0.648779)	0.500331 / 4.584777 (-4.084446)	3.113662 / 3.745712 (-0.632050)	2.931540 / 5.269862 (-2.338322)	1.853615 / 4.565676 (-2.712062)	0.058250 / 0.424275 (-0.366025)	0.006546 / 0.007607 (-0.001061)	0.508850 / 0.226044 (0.282806)	5.081809 / 2.268929 (2.812880)	2.687037 / 55.444624 (-52.757588)	2.369317 / 6.876477 (-4.507160)	2.383549 / 2.142072 (0.241477)	0.587039 / 4.805227 (-4.218188)	0.125858 / 6.500664 (-6.374806)	0.062522 / 0.075469 (-0.012947)

Benchmark: benchmark_map_filter.json

metric	filter	map fast-tokenizer batched	map identity	map identity batched	map no-op batched	map no-op batched numpy	map no-op batched pandas	map no-op batched pytorch	map no-op batched tensorflow
new / old (diff)	1.294929 / 1.841788 (-0.546858)	18.056312 / 8.074308 (9.982004)	13.755117 / 10.191392 (3.563725)	0.132037 / 0.680424 (-0.548387)	0.016866 / 0.534201 (-0.517335)	0.339040 / 0.579283 (-0.240243)	0.364371 / 0.434364 (-0.069993)	0.399533 / 0.540337 (-0.140804)	0.564524 / 1.386936 (-0.822412)

lhoestq

Cool !

(nit) there is one progress bar without description in load_dataset. When running

ds = load_dataset("lhoestq/demo1")

I get

Downloading data files: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:00<00:00, 1440.10it/s]
Extracting data files: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:00<00:00, 1706.39it/s]
Generating train split: 5 examples [00:00, 253.97 examples/s]
Generating test split: 5 examples [00:00, 2493.64 examples/s]
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:00<00:00, 572.29it/s]

mariosasko · 2023-07-12T15:07:46Z

@lhoestq This bar comes from:

datasets/src/datasets/builder.py

Lines 1156 to 1166 in b8067c0

    
           datasets = map_nested( 
        
               partial( 
        
                   self._build_single_dataset, 
        
                   run_post_process=run_post_process, 
        
                   verification_mode=verification_mode, 
        
                   in_memory=in_memory, 
        
               ), 
        
               split, 
        
               map_tuple=True, 
        
               disable_tqdm=not logging.is_progress_bar_enabled(), 
        
           )

Do you prefer not showing it or, e.g., having desc="Generating splits"?

lhoestq · 2023-07-12T15:14:16Z

No strong opinion. Since there is a "Generating" progress bar already, maybe it can be "Preparing splits" (ref to download_and_prepare)

github-actions · 2023-07-12T15:57:09Z

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

Benchmark: benchmark_array_xd.json

metric	read_batch_formatted_as_numpy after write_array2d	read_batch_formatted_as_numpy after write_flattened_sequence	read_batch_formatted_as_numpy after write_nested_sequence	read_batch_unformated after write_array2d	read_batch_unformated after write_flattened_sequence	read_batch_unformated after write_nested_sequence	read_col_formatted_as_numpy after write_array2d	read_col_formatted_as_numpy after write_flattened_sequence	read_col_formatted_as_numpy after write_nested_sequence	read_col_unformated after write_array2d	read_col_unformated after write_flattened_sequence	read_col_unformated after write_nested_sequence	read_formatted_as_numpy after write_array2d	read_formatted_as_numpy after write_flattened_sequence	read_formatted_as_numpy after write_nested_sequence	read_unformated after write_array2d	read_unformated after write_flattened_sequence	read_unformated after write_nested_sequence	write_array2d	write_flattened_sequence	write_nested_sequence
new / old (diff)	0.006348 / 0.011353 (-0.005005)	0.003721 / 0.011008 (-0.007287)	0.084039 / 0.038508 (0.045531)	0.067627 / 0.023109 (0.044517)	0.308372 / 0.275898 (0.032474)	0.335131 / 0.323480 (0.011652)	0.005157 / 0.007986 (-0.002829)	0.003266 / 0.004328 (-0.001062)	0.065374 / 0.004250 (0.061124)	0.055550 / 0.037052 (0.018498)	0.314001 / 0.258489 (0.055512)	0.350510 / 0.293841 (0.056669)	0.030859 / 0.128546 (-0.097688)	0.008286 / 0.075646 (-0.067361)	0.287122 / 0.419271 (-0.132149)	0.051494 / 0.043533 (0.007961)	0.309868 / 0.255139 (0.054729)	0.325845 / 0.283200 (0.042645)	0.022622 / 0.141683 (-0.119061)	1.468730 / 1.452155 (0.016575)	1.547871 / 1.492716 (0.055155)

Benchmark: benchmark_getitem_100B.json

metric	get_batch_of_1024_random_rows	get_batch_of_1024_rows	get_first_row	get_last_row
new / old (diff)	0.202763 / 0.018006 (0.184757)	0.456403 / 0.000490 (0.455914)	0.003116 / 0.000200 (0.002916)	0.000079 / 0.000054 (0.000024)

Benchmark: benchmark_indices_mapping.json

metric	select	shard	shuffle	sort	train_test_split
new / old (diff)	0.027297 / 0.037411 (-0.010114)	0.081204 / 0.014526 (0.066678)	0.094274 / 0.176557 (-0.082282)	0.154391 / 0.737135 (-0.582744)	0.094312 / 0.296338 (-0.202026)

Benchmark: benchmark_iterating.json

metric	read 5000	read 50000	read_batch 50000 10	read_batch 50000 100	read_batch 50000 1000	read_formatted numpy 5000	read_formatted pandas 5000	read_formatted tensorflow 5000	read_formatted torch 5000	read_formatted_batch numpy 5000 10	read_formatted_batch numpy 5000 1000	shuffled read 5000	shuffled read 50000	shuffled read_batch 50000 10	shuffled read_batch 50000 100	shuffled read_batch 50000 1000	shuffled read_formatted numpy 5000	shuffled read_formatted_batch numpy 5000 10	shuffled read_formatted_batch numpy 5000 1000
new / old (diff)	0.387382 / 0.215209 (0.172173)	3.865597 / 2.077655 (1.787943)	1.855959 / 1.504120 (0.351839)	1.685411 / 1.541195 (0.144216)	1.732127 / 1.468490 (0.263637)	0.482230 / 4.584777 (-4.102547)	3.664947 / 3.745712 (-0.080765)	5.114379 / 5.269862 (-0.155482)	3.102803 / 4.565676 (-1.462873)	0.056509 / 0.424275 (-0.367766)	0.007230 / 0.007607 (-0.000377)	0.456788 / 0.226044 (0.230744)	4.575831 / 2.268929 (2.306902)	2.335249 / 55.444624 (-53.109375)	2.003805 / 6.876477 (-4.872672)	2.141788 / 2.142072 (-0.000285)	0.577501 / 4.805227 (-4.227726)	0.130264 / 6.500664 (-6.370400)	0.058889 / 0.075469 (-0.016580)

Benchmark: benchmark_map_filter.json

metric	filter	map fast-tokenizer batched	map identity	map identity batched	map no-op batched	map no-op batched numpy	map no-op batched pandas	map no-op batched pytorch	map no-op batched tensorflow
new / old (diff)	1.252673 / 1.841788 (-0.589115)	18.676897 / 8.074308 (10.602589)	13.988101 / 10.191392 (3.796709)	0.151376 / 0.680424 (-0.529048)	0.018104 / 0.534201 (-0.516097)	0.388413 / 0.579283 (-0.190870)	0.414841 / 0.434364 (-0.019523)	0.456078 / 0.540337 (-0.084259)	0.641715 / 1.386936 (-0.745221)

PyArrow==latest

Show updated benchmarks!

Benchmark: benchmark_array_xd.json

metric	read_batch_formatted_as_numpy after write_array2d	read_batch_formatted_as_numpy after write_flattened_sequence	read_batch_formatted_as_numpy after write_nested_sequence	read_batch_unformated after write_array2d	read_batch_unformated after write_flattened_sequence	read_batch_unformated after write_nested_sequence	read_col_formatted_as_numpy after write_array2d	read_col_formatted_as_numpy after write_flattened_sequence	read_col_formatted_as_numpy after write_nested_sequence	read_col_unformated after write_array2d	read_col_unformated after write_flattened_sequence	read_col_unformated after write_nested_sequence	read_formatted_as_numpy after write_array2d	read_formatted_as_numpy after write_flattened_sequence	read_formatted_as_numpy after write_nested_sequence	read_unformated after write_array2d	read_unformated after write_flattened_sequence	read_unformated after write_nested_sequence	write_array2d	write_flattened_sequence	write_nested_sequence
new / old (diff)	0.006315 / 0.011353 (-0.005038)	0.003847 / 0.011008 (-0.007162)	0.063989 / 0.038508 (0.025481)	0.068244 / 0.023109 (0.045135)	0.416201 / 0.275898 (0.140303)	0.438446 / 0.323480 (0.114966)	0.005820 / 0.007986 (-0.002166)	0.003165 / 0.004328 (-0.001163)	0.064143 / 0.004250 (0.059892)	0.056529 / 0.037052 (0.019477)	0.414916 / 0.258489 (0.156427)	0.450771 / 0.293841 (0.156930)	0.030611 / 0.128546 (-0.097935)	0.008289 / 0.075646 (-0.067357)	0.070725 / 0.419271 (-0.348546)	0.047998 / 0.043533 (0.004465)	0.405609 / 0.255139 (0.150470)	0.421895 / 0.283200 (0.138696)	0.022135 / 0.141683 (-0.119548)	1.444238 / 1.452155 (-0.007916)	1.515823 / 1.492716 (0.023107)

Benchmark: benchmark_getitem_100B.json

metric	get_batch_of_1024_random_rows	get_batch_of_1024_rows	get_first_row	get_last_row
new / old (diff)	0.227043 / 0.018006 (0.209037)	0.439732 / 0.000490 (0.439242)	0.001267 / 0.000200 (0.001067)	0.000070 / 0.000054 (0.000016)

Benchmark: benchmark_indices_mapping.json

metric	select	shard	shuffle	sort	train_test_split
new / old (diff)	0.029082 / 0.037411 (-0.008329)	0.086201 / 0.014526 (0.071675)	0.098653 / 0.176557 (-0.077903)	0.152574 / 0.737135 (-0.584561)	0.100696 / 0.296338 (-0.195642)

Benchmark: benchmark_iterating.json

metric	read 5000	read 50000	read_batch 50000 10	read_batch 50000 100	read_batch 50000 1000	read_formatted numpy 5000	read_formatted pandas 5000	read_formatted tensorflow 5000	read_formatted torch 5000	read_formatted_batch numpy 5000 10	read_formatted_batch numpy 5000 1000	shuffled read 5000	shuffled read 50000	shuffled read_batch 50000 10	shuffled read_batch 50000 100	shuffled read_batch 50000 1000	shuffled read_formatted numpy 5000	shuffled read_formatted_batch numpy 5000 10	shuffled read_formatted_batch numpy 5000 1000
new / old (diff)	0.411243 / 0.215209 (0.196034)	4.100170 / 2.077655 (2.022515)	2.118310 / 1.504120 (0.614190)	1.935646 / 1.541195 (0.394451)	1.970798 / 1.468490 (0.502307)	0.478635 / 4.584777 (-4.106142)	3.589396 / 3.745712 (-0.156316)	3.312462 / 5.269862 (-1.957399)	1.963081 / 4.565676 (-2.602595)	0.056392 / 0.424275 (-0.367883)	0.007134 / 0.007607 (-0.000473)	0.485131 / 0.226044 (0.259086)	4.838946 / 2.268929 (2.570017)	2.624550 / 55.444624 (-52.820075)	2.223046 / 6.876477 (-4.653431)	2.230642 / 2.142072 (0.088570)	0.594892 / 4.805227 (-4.210335)	0.130523 / 6.500664 (-6.370141)	0.059585 / 0.075469 (-0.015884)

Benchmark: benchmark_map_filter.json

metric	filter	map fast-tokenizer batched	map identity	map identity batched	map no-op batched	map no-op batched numpy	map no-op batched pandas	map no-op batched pytorch	map no-op batched tensorflow
new / old (diff)	1.329941 / 1.841788 (-0.511847)	19.199057 / 8.074308 (11.124748)	14.166009 / 10.191392 (3.974617)	0.190595 / 0.680424 (-0.489829)	0.018419 / 0.534201 (-0.515782)	0.392031 / 0.579283 (-0.187252)	0.409395 / 0.434364 (-0.024969)	0.475930 / 0.540337 (-0.064408)	0.654412 / 1.386936 (-0.732524)

github-actions · 2023-07-12T16:26:44Z

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

Benchmark: benchmark_array_xd.json

metric	read_batch_formatted_as_numpy after write_array2d	read_batch_formatted_as_numpy after write_flattened_sequence	read_batch_formatted_as_numpy after write_nested_sequence	read_batch_unformated after write_array2d	read_batch_unformated after write_flattened_sequence	read_batch_unformated after write_nested_sequence	read_col_formatted_as_numpy after write_array2d	read_col_formatted_as_numpy after write_flattened_sequence	read_col_formatted_as_numpy after write_nested_sequence	read_col_unformated after write_array2d	read_col_unformated after write_flattened_sequence	read_col_unformated after write_nested_sequence	read_formatted_as_numpy after write_array2d	read_formatted_as_numpy after write_flattened_sequence	read_formatted_as_numpy after write_nested_sequence	read_unformated after write_array2d	read_unformated after write_flattened_sequence	read_unformated after write_nested_sequence	write_array2d	write_flattened_sequence	write_nested_sequence
new / old (diff)	0.007500 / 0.011353 (-0.003853)	0.004328 / 0.011008 (-0.006681)	0.086718 / 0.038508 (0.048209)	0.098638 / 0.023109 (0.075529)	0.335308 / 0.275898 (0.059409)	0.369163 / 0.323480 (0.045683)	0.005733 / 0.007986 (-0.002253)	0.003738 / 0.004328 (-0.000590)	0.066452 / 0.004250 (0.062202)	0.066245 / 0.037052 (0.029192)	0.337609 / 0.258489 (0.079120)	0.388584 / 0.293841 (0.094744)	0.031742 / 0.128546 (-0.096804)	0.008721 / 0.075646 (-0.066925)	0.290820 / 0.419271 (-0.128452)	0.053323 / 0.043533 (0.009790)	0.329192 / 0.255139 (0.074053)	0.350560 / 0.283200 (0.067360)	0.025402 / 0.141683 (-0.116281)	1.476174 / 1.452155 (0.024020)	1.578194 / 1.492716 (0.085478)

Benchmark: benchmark_getitem_100B.json

metric	get_batch_of_1024_random_rows	get_batch_of_1024_rows	get_first_row	get_last_row
new / old (diff)	0.256160 / 0.018006 (0.238154)	0.560315 / 0.000490 (0.559825)	0.005287 / 0.000200 (0.005088)	0.000094 / 0.000054 (0.000040)

Benchmark: benchmark_indices_mapping.json

metric	select	shard	shuffle	sort	train_test_split
new / old (diff)	0.029164 / 0.037411 (-0.008247)	0.084881 / 0.014526 (0.070356)	0.100979 / 0.176557 (-0.075577)	0.156539 / 0.737135 (-0.580597)	0.101510 / 0.296338 (-0.194828)

Benchmark: benchmark_iterating.json

metric	read 5000	read 50000	read_batch 50000 10	read_batch 50000 100	read_batch 50000 1000	read_formatted numpy 5000	read_formatted pandas 5000	read_formatted tensorflow 5000	read_formatted torch 5000	read_formatted_batch numpy 5000 10	read_formatted_batch numpy 5000 1000	shuffled read 5000	shuffled read 50000	shuffled read_batch 50000 10	shuffled read_batch 50000 100	shuffled read_batch 50000 1000	shuffled read_formatted numpy 5000	shuffled read_formatted_batch numpy 5000 10	shuffled read_formatted_batch numpy 5000 1000
new / old (diff)	0.381138 / 0.215209 (0.165929)	3.791573 / 2.077655 (1.713918)	1.841954 / 1.504120 (0.337834)	1.672463 / 1.541195 (0.131268)	1.785769 / 1.468490 (0.317279)	0.483263 / 4.584777 (-4.101514)	3.617391 / 3.745712 (-0.128322)	5.607794 / 5.269862 (0.337933)	3.359530 / 4.565676 (-1.206147)	0.056826 / 0.424275 (-0.367449)	0.007375 / 0.007607 (-0.000232)	0.455853 / 0.226044 (0.229809)	4.548965 / 2.268929 (2.280037)	2.412716 / 55.444624 (-53.031908)	1.991456 / 6.876477 (-4.885021)	2.242851 / 2.142072 (0.100778)	0.573070 / 4.805227 (-4.232157)	0.134658 / 6.500664 (-6.366006)	0.061539 / 0.075469 (-0.013930)

Benchmark: benchmark_map_filter.json

metric	filter	map fast-tokenizer batched	map identity	map identity batched	map no-op batched	map no-op batched numpy	map no-op batched pandas	map no-op batched pytorch	map no-op batched tensorflow
new / old (diff)	1.278306 / 1.841788 (-0.563481)	20.634317 / 8.074308 (12.560009)	15.164246 / 10.191392 (4.972854)	0.167487 / 0.680424 (-0.512937)	0.019006 / 0.534201 (-0.515195)	0.394617 / 0.579283 (-0.184666)	0.423385 / 0.434364 (-0.010979)	0.469968 / 0.540337 (-0.070370)	0.630058 / 1.386936 (-0.756878)

PyArrow==latest

Show updated benchmarks!

Benchmark: benchmark_array_xd.json

metric	read_batch_formatted_as_numpy after write_array2d	read_batch_formatted_as_numpy after write_flattened_sequence	read_batch_formatted_as_numpy after write_nested_sequence	read_batch_unformated after write_array2d	read_batch_unformated after write_flattened_sequence	read_batch_unformated after write_nested_sequence	read_col_formatted_as_numpy after write_array2d	read_col_formatted_as_numpy after write_flattened_sequence	read_col_formatted_as_numpy after write_nested_sequence	read_col_unformated after write_array2d	read_col_unformated after write_flattened_sequence	read_col_unformated after write_nested_sequence	read_formatted_as_numpy after write_array2d	read_formatted_as_numpy after write_flattened_sequence	read_formatted_as_numpy after write_nested_sequence	read_unformated after write_array2d	read_unformated after write_flattened_sequence	read_unformated after write_nested_sequence	write_array2d	write_flattened_sequence	write_nested_sequence
new / old (diff)	0.006793 / 0.011353 (-0.004559)	0.004260 / 0.011008 (-0.006748)	0.065398 / 0.038508 (0.026890)	0.077850 / 0.023109 (0.054741)	0.371754 / 0.275898 (0.095855)	0.400652 / 0.323480 (0.077172)	0.005729 / 0.007986 (-0.002256)	0.003660 / 0.004328 (-0.000669)	0.065119 / 0.004250 (0.060869)	0.060714 / 0.037052 (0.023661)	0.384592 / 0.258489 (0.126103)	0.412806 / 0.293841 (0.118965)	0.031865 / 0.128546 (-0.096681)	0.008807 / 0.075646 (-0.066839)	0.071156 / 0.419271 (-0.348115)	0.049571 / 0.043533 (0.006038)	0.367381 / 0.255139 (0.112242)	0.386713 / 0.283200 (0.103513)	0.024838 / 0.141683 (-0.116845)	1.492986 / 1.452155 (0.040831)	1.559243 / 1.492716 (0.066526)

Benchmark: benchmark_getitem_100B.json

metric	get_batch_of_1024_random_rows	get_batch_of_1024_rows	get_first_row	get_last_row
new / old (diff)	0.269737 / 0.018006 (0.251730)	0.565177 / 0.000490 (0.564687)	0.000404 / 0.000200 (0.000204)	0.000060 / 0.000054 (0.000006)

Benchmark: benchmark_indices_mapping.json

metric	select	shard	shuffle	sort	train_test_split
new / old (diff)	0.031631 / 0.037411 (-0.005780)	0.087289 / 0.014526 (0.072764)	0.102798 / 0.176557 (-0.073759)	0.158977 / 0.737135 (-0.578158)	0.105495 / 0.296338 (-0.190843)

Benchmark: benchmark_iterating.json

metric	read 5000	read 50000	read_batch 50000 10	read_batch 50000 100	read_batch 50000 1000	read_formatted numpy 5000	read_formatted pandas 5000	read_formatted tensorflow 5000	read_formatted torch 5000	read_formatted_batch numpy 5000 10	read_formatted_batch numpy 5000 1000	shuffled read 5000	shuffled read 50000	shuffled read_batch 50000 10	shuffled read_batch 50000 100	shuffled read_batch 50000 1000	shuffled read_formatted numpy 5000	shuffled read_formatted_batch numpy 5000 10	shuffled read_formatted_batch numpy 5000 1000
new / old (diff)	0.425067 / 0.215209 (0.209858)	4.243121 / 2.077655 (2.165466)	2.234567 / 1.504120 (0.730447)	2.070810 / 1.541195 (0.529615)	2.176802 / 1.468490 (0.708312)	0.484987 / 4.584777 (-4.099790)	3.647000 / 3.745712 (-0.098712)	3.574843 / 5.269862 (-1.695019)	2.092581 / 4.565676 (-2.473095)	0.057299 / 0.424275 (-0.366976)	0.007480 / 0.007607 (-0.000128)	0.507838 / 0.226044 (0.281794)	5.076594 / 2.268929 (2.807666)	2.718858 / 55.444624 (-52.725766)	2.362793 / 6.876477 (-4.513684)	2.451962 / 2.142072 (0.309890)	0.581355 / 4.805227 (-4.223872)	0.133723 / 6.500664 (-6.366941)	0.061896 / 0.075469 (-0.013573)

Benchmark: benchmark_map_filter.json

metric	filter	map fast-tokenizer batched	map identity	map identity batched	map no-op batched	map no-op batched numpy	map no-op batched pandas	map no-op batched pytorch	map no-op batched tensorflow
new / old (diff)	1.325814 / 1.841788 (-0.515974)	20.614502 / 8.074308 (12.540194)	14.769422 / 10.191392 (4.578029)	0.193797 / 0.680424 (-0.486627)	0.018379 / 0.534201 (-0.515822)	0.394153 / 0.579283 (-0.185130)	0.409585 / 0.434364 (-0.024779)	0.479107 / 0.540337 (-0.061231)	0.668397 / 1.386936 (-0.718539)

mariosasko · 2023-07-12T17:19:20Z

In the end, I decided to remove the progress bar to avoid having it displayed when loading a cached dataset.

github-actions · 2023-07-12T17:28:25Z

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

Benchmark: benchmark_array_xd.json

metric	read_batch_formatted_as_numpy after write_array2d	read_batch_formatted_as_numpy after write_flattened_sequence	read_batch_formatted_as_numpy after write_nested_sequence	read_batch_unformated after write_array2d	read_batch_unformated after write_flattened_sequence	read_batch_unformated after write_nested_sequence	read_col_formatted_as_numpy after write_array2d	read_col_formatted_as_numpy after write_flattened_sequence	read_col_formatted_as_numpy after write_nested_sequence	read_col_unformated after write_array2d	read_col_unformated after write_flattened_sequence	read_col_unformated after write_nested_sequence	read_formatted_as_numpy after write_array2d	read_formatted_as_numpy after write_flattened_sequence	read_formatted_as_numpy after write_nested_sequence	read_unformated after write_array2d	read_unformated after write_flattened_sequence	read_unformated after write_nested_sequence	write_array2d	write_flattened_sequence	write_nested_sequence
new / old (diff)	0.006673 / 0.011353 (-0.004680)	0.004162 / 0.011008 (-0.006846)	0.084017 / 0.038508 (0.045509)	0.079536 / 0.023109 (0.056426)	0.313594 / 0.275898 (0.037695)	0.349200 / 0.323480 (0.025720)	0.005544 / 0.007986 (-0.002441)	0.003472 / 0.004328 (-0.000857)	0.064742 / 0.004250 (0.060491)	0.056857 / 0.037052 (0.019805)	0.318635 / 0.258489 (0.060146)	0.354378 / 0.293841 (0.060537)	0.030856 / 0.128546 (-0.097690)	0.008759 / 0.075646 (-0.066887)	0.287760 / 0.419271 (-0.131511)	0.052307 / 0.043533 (0.008775)	0.316396 / 0.255139 (0.061257)	0.351408 / 0.283200 (0.068208)	0.024914 / 0.141683 (-0.116769)	1.484592 / 1.452155 (0.032437)	1.560662 / 1.492716 (0.067945)

Benchmark: benchmark_getitem_100B.json

metric	get_batch_of_1024_random_rows	get_batch_of_1024_rows	get_first_row	get_last_row
new / old (diff)	0.280938 / 0.018006 (0.262932)	0.580236 / 0.000490 (0.579747)	0.003369 / 0.000200 (0.003169)	0.000090 / 0.000054 (0.000036)

Benchmark: benchmark_indices_mapping.json

metric	select	shard	shuffle	sort	train_test_split
new / old (diff)	0.028736 / 0.037411 (-0.008675)	0.082916 / 0.014526 (0.068390)	0.097761 / 0.176557 (-0.078796)	0.153515 / 0.737135 (-0.583620)	0.099282 / 0.296338 (-0.197057)

Benchmark: benchmark_iterating.json

metric	read 5000	read 50000	read_batch 50000 10	read_batch 50000 100	read_batch 50000 1000	read_formatted numpy 5000	read_formatted pandas 5000	read_formatted tensorflow 5000	read_formatted torch 5000	read_formatted_batch numpy 5000 10	read_formatted_batch numpy 5000 1000	shuffled read 5000	shuffled read 50000	shuffled read_batch 50000 10	shuffled read_batch 50000 100	shuffled read_batch 50000 1000	shuffled read_formatted numpy 5000	shuffled read_formatted_batch numpy 5000 10	shuffled read_formatted_batch numpy 5000 1000
new / old (diff)	0.401244 / 0.215209 (0.186035)	4.019866 / 2.077655 (1.942211)	2.029642 / 1.504120 (0.525522)	1.849591 / 1.541195 (0.308396)	1.946829 / 1.468490 (0.478339)	0.479750 / 4.584777 (-4.105027)	3.482822 / 3.745712 (-0.262890)	3.955859 / 5.269862 (-1.314003)	2.370747 / 4.565676 (-2.194930)	0.056905 / 0.424275 (-0.367370)	0.007319 / 0.007607 (-0.000288)	0.485310 / 0.226044 (0.259266)	4.858228 / 2.268929 (2.589299)	2.500476 / 55.444624 (-52.944148)	2.171156 / 6.876477 (-4.705320)	2.427266 / 2.142072 (0.285194)	0.570199 / 4.805227 (-4.235029)	0.130855 / 6.500664 (-6.369809)	0.060269 / 0.075469 (-0.015200)

Benchmark: benchmark_map_filter.json

metric	filter	map fast-tokenizer batched	map identity	map identity batched	map no-op batched	map no-op batched numpy	map no-op batched pandas	map no-op batched pytorch	map no-op batched tensorflow
new / old (diff)	1.258044 / 1.841788 (-0.583743)	20.218657 / 8.074308 (12.144349)	13.597970 / 10.191392 (3.406578)	0.167656 / 0.680424 (-0.512768)	0.018137 / 0.534201 (-0.516064)	0.395309 / 0.579283 (-0.183975)	0.406325 / 0.434364 (-0.028039)	0.467457 / 0.540337 (-0.072880)	0.613636 / 1.386936 (-0.773300)

PyArrow==latest

Show updated benchmarks!

Benchmark: benchmark_array_xd.json

metric	read_batch_formatted_as_numpy after write_array2d	read_batch_formatted_as_numpy after write_flattened_sequence	read_batch_formatted_as_numpy after write_nested_sequence	read_batch_unformated after write_array2d	read_batch_unformated after write_flattened_sequence	read_batch_unformated after write_nested_sequence	read_col_formatted_as_numpy after write_array2d	read_col_formatted_as_numpy after write_flattened_sequence	read_col_formatted_as_numpy after write_nested_sequence	read_col_unformated after write_array2d	read_col_unformated after write_flattened_sequence	read_col_unformated after write_nested_sequence	read_formatted_as_numpy after write_array2d	read_formatted_as_numpy after write_flattened_sequence	read_formatted_as_numpy after write_nested_sequence	read_unformated after write_array2d	read_unformated after write_flattened_sequence	read_unformated after write_nested_sequence	write_array2d	write_flattened_sequence	write_nested_sequence
new / old (diff)	0.006846 / 0.011353 (-0.004507)	0.004207 / 0.011008 (-0.006802)	0.064525 / 0.038508 (0.026017)	0.081329 / 0.023109 (0.058220)	0.399838 / 0.275898 (0.123940)	0.431305 / 0.323480 (0.107825)	0.005859 / 0.007986 (-0.002127)	0.003568 / 0.004328 (-0.000760)	0.065262 / 0.004250 (0.061011)	0.064796 / 0.037052 (0.027744)	0.406858 / 0.258489 (0.148369)	0.440971 / 0.293841 (0.147130)	0.031421 / 0.128546 (-0.097125)	0.008777 / 0.075646 (-0.066870)	0.071418 / 0.419271 (-0.347853)	0.049263 / 0.043533 (0.005730)	0.384279 / 0.255139 (0.129140)	0.410745 / 0.283200 (0.127546)	0.024467 / 0.141683 (-0.117216)	1.522379 / 1.452155 (0.070224)	1.581636 / 1.492716 (0.088920)

Benchmark: benchmark_getitem_100B.json

metric	get_batch_of_1024_random_rows	get_batch_of_1024_rows	get_first_row	get_last_row
new / old (diff)	0.276161 / 0.018006 (0.258155)	0.548842 / 0.000490 (0.548352)	0.004523 / 0.000200 (0.004324)	0.000098 / 0.000054 (0.000043)

Benchmark: benchmark_indices_mapping.json

metric	select	shard	shuffle	sort	train_test_split
new / old (diff)	0.030747 / 0.037411 (-0.006664)	0.087493 / 0.014526 (0.072967)	0.106563 / 0.176557 (-0.069993)	0.162949 / 0.737135 (-0.574186)	0.105303 / 0.296338 (-0.191036)

Benchmark: benchmark_iterating.json

metric	read 5000	read 50000	read_batch 50000 10	read_batch 50000 100	read_batch 50000 1000	read_formatted numpy 5000	read_formatted pandas 5000	read_formatted tensorflow 5000	read_formatted torch 5000	read_formatted_batch numpy 5000 10	read_formatted_batch numpy 5000 1000	shuffled read 5000	shuffled read 50000	shuffled read_batch 50000 10	shuffled read_batch 50000 100	shuffled read_batch 50000 1000	shuffled read_formatted numpy 5000	shuffled read_formatted_batch numpy 5000 10	shuffled read_formatted_batch numpy 5000 1000
new / old (diff)	0.425854 / 0.215209 (0.210645)	4.244797 / 2.077655 (2.167142)	2.269006 / 1.504120 (0.764886)	2.097428 / 1.541195 (0.556234)	2.181038 / 1.468490 (0.712548)	0.477286 / 4.584777 (-4.107491)	3.591452 / 3.745712 (-0.154260)	3.481281 / 5.269862 (-1.788580)	2.066895 / 4.565676 (-2.498782)	0.056576 / 0.424275 (-0.367699)	0.007409 / 0.007607 (-0.000199)	0.498411 / 0.226044 (0.272367)	4.994873 / 2.268929 (2.725945)	2.749148 / 55.444624 (-52.695476)	2.378544 / 6.876477 (-4.497932)	2.452859 / 2.142072 (0.310786)	0.571340 / 4.805227 (-4.233887)	0.132174 / 6.500664 (-6.368490)	0.061507 / 0.075469 (-0.013962)

Benchmark: benchmark_map_filter.json

metric	filter	map fast-tokenizer batched	map identity	map identity batched	map no-op batched	map no-op batched numpy	map no-op batched pandas	map no-op batched pytorch	map no-op batched tensorflow
new / old (diff)	1.370773 / 1.841788 (-0.471015)	20.493342 / 8.074308 (12.419034)	14.809886 / 10.191392 (4.618494)	0.175730 / 0.680424 (-0.504693)	0.018617 / 0.534201 (-0.515583)	0.393808 / 0.579283 (-0.185476)	0.416419 / 0.434364 (-0.017945)	0.477183 / 0.540337 (-0.063155)	0.668060 / 1.386936 (-0.718876)

davidgilbertson · 2023-07-12T19:34:14Z

Nice one :)

mariosasko added 2 commits July 11, 2023 20:27

Improve logging

1cb7ae5

Typo

0160475

Nit

b2fc21e

Fix tests

b85b115

Fix

64b811c

mariosasko marked this pull request as ready for review July 12, 2023 11:41

mariosasko requested a review from lhoestq July 12, 2023 11:41

lhoestq approved these changes Jul 12, 2023

View reviewed changes

Nit

42fdfbd

Remove bar

b2d8922

mariosasko merged commit 2de7a2a into main Jul 12, 2023

mariosasko deleted the improve-logging branch July 12, 2023 17:19

mariosasko mentioned this pull request Jul 12, 2023

Modify levels of some logging messages #5934

Closed

mariosasko mentioned this pull request Jul 21, 2023

Make all print statements optional #5647

Closed

Improve logging #6019

Improve logging #6019

Conversation

mariosasko commented Jul 11, 2023 • edited Loading

HuggingFaceDocBuilderDev commented Jul 11, 2023 • edited Loading

github-actions bot commented Jul 11, 2023

Benchmark: benchmark_array_xd.json

Benchmark: benchmark_getitem_100B.json

Benchmark: benchmark_indices_mapping.json

Benchmark: benchmark_iterating.json

Benchmark: benchmark_map_filter.json

Benchmark: benchmark_array_xd.json

Benchmark: benchmark_getitem_100B.json

Benchmark: benchmark_indices_mapping.json

Benchmark: benchmark_iterating.json

Benchmark: benchmark_map_filter.json

github-actions bot commented Jul 11, 2023

Benchmark: benchmark_array_xd.json

Benchmark: benchmark_getitem_100B.json

Benchmark: benchmark_indices_mapping.json

Benchmark: benchmark_iterating.json

Benchmark: benchmark_map_filter.json

Benchmark: benchmark_array_xd.json

Benchmark: benchmark_getitem_100B.json

Benchmark: benchmark_indices_mapping.json

Benchmark: benchmark_iterating.json

Benchmark: benchmark_map_filter.json

github-actions bot commented Jul 11, 2023

Benchmark: benchmark_array_xd.json

Benchmark: benchmark_getitem_100B.json

Benchmark: benchmark_indices_mapping.json

Benchmark: benchmark_iterating.json

Benchmark: benchmark_map_filter.json

Benchmark: benchmark_array_xd.json

Benchmark: benchmark_getitem_100B.json

Benchmark: benchmark_indices_mapping.json

Benchmark: benchmark_iterating.json

Benchmark: benchmark_map_filter.json

github-actions bot commented Jul 11, 2023

Benchmark: benchmark_array_xd.json

Benchmark: benchmark_getitem_100B.json

Benchmark: benchmark_indices_mapping.json

Benchmark: benchmark_iterating.json

Benchmark: benchmark_map_filter.json

Benchmark: benchmark_array_xd.json

Benchmark: benchmark_getitem_100B.json

Benchmark: benchmark_indices_mapping.json

Benchmark: benchmark_iterating.json

Benchmark: benchmark_map_filter.json

github-actions bot commented Jul 11, 2023

Benchmark: benchmark_array_xd.json

Benchmark: benchmark_getitem_100B.json

Benchmark: benchmark_indices_mapping.json

Benchmark: benchmark_iterating.json

Benchmark: benchmark_map_filter.json

Benchmark: benchmark_array_xd.json

Benchmark: benchmark_getitem_100B.json

Benchmark: benchmark_indices_mapping.json

Benchmark: benchmark_iterating.json

Benchmark: benchmark_map_filter.json

lhoestq left a comment

Choose a reason for hiding this comment

mariosasko commented Jul 12, 2023

lhoestq commented Jul 12, 2023

github-actions bot commented Jul 12, 2023

Benchmark: benchmark_array_xd.json

Benchmark: benchmark_getitem_100B.json

Benchmark: benchmark_indices_mapping.json

Benchmark: benchmark_iterating.json

Benchmark: benchmark_map_filter.json

Benchmark: benchmark_array_xd.json

Benchmark: benchmark_getitem_100B.json

Benchmark: benchmark_indices_mapping.json

Benchmark: benchmark_iterating.json

Benchmark: benchmark_map_filter.json

github-actions bot commented Jul 12, 2023

Benchmark: benchmark_array_xd.json

Benchmark: benchmark_getitem_100B.json

Benchmark: benchmark_indices_mapping.json

Benchmark: benchmark_iterating.json

mariosasko commented Jul 11, 2023 •

edited

Loading

HuggingFaceDocBuilderDev commented Jul 11, 2023 •

edited

Loading