Add accelerate as metric's test dependency to fix CI error

[mariosasko]

The frugalscore metric uses Transformers' Trainer, which requires accelerate (as of recently).

Fixes the following CI error.

[HuggingFaceDocBuilderDev]

The documentation is not available anymore as the PR was closed or merged.

[github-actions]

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

Benchmark: benchmark_array_xd.json

metric	read_batch_formatted_as_numpy after write_array2d	read_batch_formatted_as_numpy after write_flattened_sequence	read_batch_formatted_as_numpy after write_nested_sequence	read_batch_unformated after write_array2d	read_batch_unformated after write_flattened_sequence	read_batch_unformated after write_nested_sequence	read_col_formatted_as_numpy after write_array2d	read_col_formatted_as_numpy after write_flattened_sequence	read_col_formatted_as_numpy after write_nested_sequence	read_col_unformated after write_array2d	read_col_unformated after write_flattened_sequence	read_col_unformated after write_nested_sequence	read_formatted_as_numpy after write_array2d	read_formatted_as_numpy after write_flattened_sequence	read_formatted_as_numpy after write_nested_sequence	read_unformated after write_array2d	read_unformated after write_flattened_sequence	read_unformated after write_nested_sequence	write_array2d	write_flattened_sequence	write_nested_sequence
new / old (diff)	0.007565 / 0.011353 (-0.003788)	0.005361 / 0.011008 (-0.005647)	0.098963 / 0.038508 (0.060455)	0.034271 / 0.023109 (0.011162)	0.323421 / 0.275898 (0.047523)	0.348495 / 0.323480 (0.025015)	0.006244 / 0.007986 (-0.001741)	0.004215 / 0.004328 (-0.000113)	0.073614 / 0.004250 (0.069364)	0.049334 / 0.037052 (0.012282)	0.315277 / 0.258489 (0.056788)	0.354325 / 0.293841 (0.060484)	0.035001 / 0.128546 (-0.093545)	0.012149 / 0.075646 (-0.063497)	0.335614 / 0.419271 (-0.083657)	0.050532 / 0.043533 (0.006999)	0.308500 / 0.255139 (0.053361)	0.324620 / 0.283200 (0.041421)	0.110241 / 0.141683 (-0.031442)	1.443923 / 1.452155 (-0.008232)	1.559289 / 1.492716 (0.066573)

Benchmark: benchmark_getitem_100B.json

metric	get_batch_of_1024_random_rows	get_batch_of_1024_rows	get_first_row	get_last_row
new / old (diff)	0.207629 / 0.018006 (0.189622)	0.433251 / 0.000490 (0.432762)	0.003021 / 0.000200 (0.002821)	0.000074 / 0.000054 (0.000019)

Benchmark: benchmark_indices_mapping.json

metric	select	shard	shuffle	sort	train_test_split
new / old (diff)	0.028312 / 0.037411 (-0.009100)	0.111829 / 0.014526 (0.097303)	0.127099 / 0.176557 (-0.049458)	0.184702 / 0.737135 (-0.552433)	0.125062 / 0.296338 (-0.171277)

Benchmark: benchmark_iterating.json

metric	read 5000	read 50000	read_batch 50000 10	read_batch 50000 100	read_batch 50000 1000	read_formatted numpy 5000	read_formatted pandas 5000	read_formatted tensorflow 5000	read_formatted torch 5000	read_formatted_batch numpy 5000 10	read_formatted_batch numpy 5000 1000	shuffled read 5000	shuffled read 50000	shuffled read_batch 50000 10	shuffled read_batch 50000 100	shuffled read_batch 50000 1000	shuffled read_formatted numpy 5000	shuffled read_formatted_batch numpy 5000 10	shuffled read_formatted_batch numpy 5000 1000
new / old (diff)	0.399451 / 0.215209 (0.184242)	3.966528 / 2.077655 (1.888874)	1.826004 / 1.504120 (0.321884)	1.669547 / 1.541195 (0.128353)	1.751584 / 1.468490 (0.283094)	0.688308 / 4.584777 (-3.896469)	3.813275 / 3.745712 (0.067562)	3.181554 / 5.269862 (-2.088307)	1.750566 / 4.565676 (-2.815111)	0.085038 / 0.424275 (-0.339237)	0.011992 / 0.007607 (0.004385)	0.502374 / 0.226044 (0.276330)	4.970614 / 2.268929 (2.701686)	2.309617 / 55.444624 (-53.135007)	2.012427 / 6.876477 (-4.864050)	2.156348 / 2.142072 (0.014276)	0.834415 / 4.805227 (-3.970812)	0.167912 / 6.500664 (-6.332752)	0.065711 / 0.075469 (-0.009758)

Benchmark: benchmark_map_filter.json

metric	filter	map fast-tokenizer batched	map identity	map identity batched	map no-op batched	map no-op batched numpy	map no-op batched pandas	map no-op batched pytorch	map no-op batched tensorflow
new / old (diff)	1.223132 / 1.841788 (-0.618656)	15.126753 / 8.074308 (7.052445)	14.829184 / 10.191392 (4.637792)	0.142582 / 0.680424 (-0.537842)	0.017483 / 0.534201 (-0.516718)	0.429768 / 0.579283 (-0.149516)	0.422745 / 0.434364 (-0.011619)	0.508813 / 0.540337 (-0.031525)	0.618716 / 1.386936 (-0.768220)

PyArrow==latest

Show updated benchmarks!

Benchmark: benchmark_array_xd.json

metric	read_batch_formatted_as_numpy after write_array2d	read_batch_formatted_as_numpy after write_flattened_sequence	read_batch_formatted_as_numpy after write_nested_sequence	read_batch_unformated after write_array2d	read_batch_unformated after write_flattened_sequence	read_batch_unformated after write_nested_sequence	read_col_formatted_as_numpy after write_array2d	read_col_formatted_as_numpy after write_flattened_sequence	read_col_formatted_as_numpy after write_nested_sequence	read_col_unformated after write_array2d	read_col_unformated after write_flattened_sequence	read_col_unformated after write_nested_sequence	read_formatted_as_numpy after write_array2d	read_formatted_as_numpy after write_flattened_sequence	read_formatted_as_numpy after write_nested_sequence	read_unformated after write_array2d	read_unformated after write_flattened_sequence	read_unformated after write_nested_sequence	write_array2d	write_flattened_sequence	write_nested_sequence
new / old (diff)	0.007749 / 0.011353 (-0.003604)	0.005433 / 0.011008 (-0.005576)	0.076223 / 0.038508 (0.037715)	0.036334 / 0.023109 (0.013225)	0.375339 / 0.275898 (0.099441)	0.413674 / 0.323480 (0.090194)	0.006207 / 0.007986 (-0.001778)	0.004085 / 0.004328 (-0.000244)	0.076154 / 0.004250 (0.071904)	0.050324 / 0.037052 (0.013271)	0.382919 / 0.258489 (0.124429)	0.442508 / 0.293841 (0.148667)	0.035951 / 0.128546 (-0.092595)	0.012067 / 0.075646 (-0.063580)	0.087649 / 0.419271 (-0.331623)	0.048786 / 0.043533 (0.005253)	0.373541 / 0.255139 (0.118402)	0.400437 / 0.283200 (0.117237)	0.102622 / 0.141683 (-0.039061)	1.472443 / 1.452155 (0.020288)	1.580178 / 1.492716 (0.087462)

Benchmark: benchmark_getitem_100B.json

metric	get_batch_of_1024_random_rows	get_batch_of_1024_rows	get_first_row	get_last_row
new / old (diff)	0.222105 / 0.018006 (0.204098)	0.445465 / 0.000490 (0.444975)	0.003671 / 0.000200 (0.003471)	0.000096 / 0.000054 (0.000041)

Benchmark: benchmark_indices_mapping.json

metric	select	shard	shuffle	sort	train_test_split
new / old (diff)	0.030808 / 0.037411 (-0.006603)	0.116687 / 0.014526 (0.102161)	0.124972 / 0.176557 (-0.051584)	0.175621 / 0.737135 (-0.561514)	0.129029 / 0.296338 (-0.167310)

Benchmark: benchmark_iterating.json

metric	read 5000	read 50000	read_batch 50000 10	read_batch 50000 100	read_batch 50000 1000	read_formatted numpy 5000	read_formatted pandas 5000	read_formatted tensorflow 5000	read_formatted torch 5000	read_formatted_batch numpy 5000 10	read_formatted_batch numpy 5000 1000	shuffled read 5000	shuffled read 50000	shuffled read_batch 50000 10	shuffled read_batch 50000 100	shuffled read_batch 50000 1000	shuffled read_formatted numpy 5000	shuffled read_formatted_batch numpy 5000 10	shuffled read_formatted_batch numpy 5000 1000
new / old (diff)	0.434627 / 0.215209 (0.219418)	4.330268 / 2.077655 (2.252613)	2.140266 / 1.504120 (0.636146)	1.960705 / 1.541195 (0.419510)	2.035949 / 1.468490 (0.567459)	0.696830 / 4.584777 (-3.887947)	3.790468 / 3.745712 (0.044756)	3.194112 / 5.269862 (-2.075750)	1.577728 / 4.565676 (-2.987948)	0.085445 / 0.424275 (-0.338830)	0.012207 / 0.007607 (0.004600)	0.555199 / 0.226044 (0.329154)	5.551539 / 2.268929 (3.282610)	2.630917 / 55.444624 (-52.813707)	2.383362 / 6.876477 (-4.493114)	2.476301 / 2.142072 (0.334229)	0.845773 / 4.805227 (-3.959455)	0.169229 / 6.500664 (-6.331435)	0.066064 / 0.075469 (-0.009405)

Benchmark: benchmark_map_filter.json

metric	filter	map fast-tokenizer batched	map identity	map identity batched	map no-op batched	map no-op batched numpy	map no-op batched pandas	map no-op batched pytorch	map no-op batched tensorflow
new / old (diff)	1.277543 / 1.841788 (-0.564245)	15.775637 / 8.074308 (7.701329)	13.528588 / 10.191392 (3.337196)	0.167428 / 0.680424 (-0.512996)	0.017581 / 0.534201 (-0.516620)	0.454472 / 0.579283 (-0.124811)	0.427987 / 0.434364 (-0.006377)	0.551512 / 0.540337 (0.011175)	0.650811 / 1.386936 (-0.736125)

[github-actions]

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

Benchmark: benchmark_array_xd.json

metric	read_batch_formatted_as_numpy after write_array2d	read_batch_formatted_as_numpy after write_flattened_sequence	read_batch_formatted_as_numpy after write_nested_sequence	read_batch_unformated after write_array2d	read_batch_unformated after write_flattened_sequence	read_batch_unformated after write_nested_sequence	read_col_formatted_as_numpy after write_array2d	read_col_formatted_as_numpy after write_flattened_sequence	read_col_formatted_as_numpy after write_nested_sequence	read_col_unformated after write_array2d	read_col_unformated after write_flattened_sequence	read_col_unformated after write_nested_sequence	read_formatted_as_numpy after write_array2d	read_formatted_as_numpy after write_flattened_sequence	read_formatted_as_numpy after write_nested_sequence	read_unformated after write_array2d	read_unformated after write_flattened_sequence	read_unformated after write_nested_sequence	write_array2d	write_flattened_sequence	write_nested_sequence
new / old (diff)	0.009800 / 0.011353 (-0.001552)	0.006443 / 0.011008 (-0.004565)	0.144137 / 0.038508 (0.105629)	0.037493 / 0.023109 (0.014383)	0.482306 / 0.275898 (0.206408)	0.467625 / 0.323480 (0.144145)	0.006812 / 0.007986 (-0.001174)	0.004810 / 0.004328 (0.000481)	0.109047 / 0.004250 (0.104796)	0.047169 / 0.037052 (0.010116)	0.451253 / 0.258489 (0.192764)	0.511339 / 0.293841 (0.217498)	0.055583 / 0.128546 (-0.072963)	0.021810 / 0.075646 (-0.053836)	0.426522 / 0.419271 (0.007250)	0.070282 / 0.043533 (0.026749)	0.469631 / 0.255139 (0.214492)	0.484951 / 0.283200 (0.201751)	0.117370 / 0.141683 (-0.024313)	1.809917 / 1.452155 (0.357763)	1.882659 / 1.492716 (0.389943)

Benchmark: benchmark_getitem_100B.json

metric	get_batch_of_1024_random_rows	get_batch_of_1024_rows	get_first_row	get_last_row
new / old (diff)	0.223843 / 0.018006 (0.205837)	0.549216 / 0.000490 (0.548726)	0.007120 / 0.000200 (0.006920)	0.000128 / 0.000054 (0.000074)

Benchmark: benchmark_indices_mapping.json

metric	select	shard	shuffle	sort	train_test_split
new / old (diff)	0.033057 / 0.037411 (-0.004354)	0.128242 / 0.014526 (0.113716)	0.140906 / 0.176557 (-0.035650)	0.213122 / 0.737135 (-0.524013)	0.148115 / 0.296338 (-0.148224)

Benchmark: benchmark_iterating.json

metric	read 5000	read 50000	read_batch 50000 10	read_batch 50000 100	read_batch 50000 1000	read_formatted numpy 5000	read_formatted pandas 5000	read_formatted tensorflow 5000	read_formatted torch 5000	read_formatted_batch numpy 5000 10	read_formatted_batch numpy 5000 1000	shuffled read 5000	shuffled read 50000	shuffled read_batch 50000 10	shuffled read_batch 50000 100	shuffled read_batch 50000 1000	shuffled read_formatted numpy 5000	shuffled read_formatted_batch numpy 5000 10	shuffled read_formatted_batch numpy 5000 1000
new / old (diff)	0.638712 / 0.215209 (0.423503)	6.383684 / 2.077655 (4.306029)	2.477020 / 1.504120 (0.972900)	2.129190 / 1.541195 (0.587996)	2.230503 / 1.468490 (0.762013)	1.367167 / 4.584777 (-3.217610)	5.570586 / 3.745712 (1.824873)	5.462857 / 5.269862 (0.192996)	2.990604 / 4.565676 (-1.575073)	0.146543 / 0.424275 (-0.277732)	0.016060 / 0.007607 (0.008453)	0.812691 / 0.226044 (0.586646)	7.928041 / 2.268929 (5.659112)	3.329494 / 55.444624 (-52.115130)	2.523452 / 6.876477 (-4.353025)	2.672374 / 2.142072 (0.530302)	1.598554 / 4.805227 (-3.206673)	0.284727 / 6.500664 (-6.215937)	0.080359 / 0.075469 (0.004889)

Benchmark: benchmark_map_filter.json

metric	filter	map fast-tokenizer batched	map identity	map identity batched	map no-op batched	map no-op batched numpy	map no-op batched pandas	map no-op batched pytorch	map no-op batched tensorflow
new / old (diff)	1.501112 / 1.841788 (-0.340675)	17.553644 / 8.074308 (9.479335)	22.704062 / 10.191392 (12.512670)	0.225575 / 0.680424 (-0.454849)	0.026531 / 0.534201 (-0.507670)	0.520129 / 0.579283 (-0.059154)	0.626220 / 0.434364 (0.191856)	0.631740 / 0.540337 (0.091403)	0.750611 / 1.386936 (-0.636325)

PyArrow==latest

Show updated benchmarks!

Benchmark: benchmark_array_xd.json

metric	read_batch_formatted_as_numpy after write_array2d	read_batch_formatted_as_numpy after write_flattened_sequence	read_batch_formatted_as_numpy after write_nested_sequence	read_batch_unformated after write_array2d	read_batch_unformated after write_flattened_sequence	read_batch_unformated after write_nested_sequence	read_col_formatted_as_numpy after write_array2d	read_col_formatted_as_numpy after write_flattened_sequence	read_col_formatted_as_numpy after write_nested_sequence	read_col_unformated after write_array2d	read_col_unformated after write_flattened_sequence	read_col_unformated after write_nested_sequence	read_formatted_as_numpy after write_array2d	read_formatted_as_numpy after write_flattened_sequence	read_formatted_as_numpy after write_nested_sequence	read_unformated after write_array2d	read_unformated after write_flattened_sequence	read_unformated after write_nested_sequence	write_array2d	write_flattened_sequence	write_nested_sequence
new / old (diff)	0.009866 / 0.011353 (-0.001487)	0.005733 / 0.011008 (-0.005275)	0.111529 / 0.038508 (0.073021)	0.042001 / 0.023109 (0.018891)	0.458578 / 0.275898 (0.182680)	0.507796 / 0.323480 (0.184316)	0.006547 / 0.007986 (-0.001438)	0.005611 / 0.004328 (0.001282)	0.115321 / 0.004250 (0.111070)	0.048741 / 0.037052 (0.011689)	0.447611 / 0.258489 (0.189122)	0.531830 / 0.293841 (0.237989)	0.052176 / 0.128546 (-0.076370)	0.022431 / 0.075646 (-0.053216)	0.120709 / 0.419271 (-0.298562)	0.067301 / 0.043533 (0.023769)	0.460577 / 0.255139 (0.205438)	0.497805 / 0.283200 (0.214605)	0.121830 / 0.141683 (-0.019853)	1.876436 / 1.452155 (0.424281)	1.983491 / 1.492716 (0.490775)

Benchmark: benchmark_getitem_100B.json

metric	get_batch_of_1024_random_rows	get_batch_of_1024_rows	get_first_row	get_last_row
new / old (diff)	0.230982 / 0.018006 (0.212976)	0.540643 / 0.000490 (0.540153)	0.004646 / 0.000200 (0.004446)	0.000131 / 0.000054 (0.000077)

Benchmark: benchmark_indices_mapping.json

metric	select	shard	shuffle	sort	train_test_split
new / old (diff)	0.034230 / 0.037411 (-0.003181)	0.136454 / 0.014526 (0.121928)	0.143370 / 0.176557 (-0.033187)	0.206752 / 0.737135 (-0.530384)	0.148722 / 0.296338 (-0.147617)

Benchmark: benchmark_iterating.json

metric	read 5000	read 50000	read_batch 50000 10	read_batch 50000 100	read_batch 50000 1000	read_formatted numpy 5000	read_formatted pandas 5000	read_formatted tensorflow 5000	read_formatted torch 5000	read_formatted_batch numpy 5000 10	read_formatted_batch numpy 5000 1000	shuffled read 5000	shuffled read 50000	shuffled read_batch 50000 10	shuffled read_batch 50000 100	shuffled read_batch 50000 1000	shuffled read_formatted numpy 5000	shuffled read_formatted_batch numpy 5000 10	shuffled read_formatted_batch numpy 5000 1000
new / old (diff)	0.704667 / 0.215209 (0.489458)	7.112079 / 2.077655 (5.034424)	3.083916 / 1.504120 (1.579797)	2.606388 / 1.541195 (1.065193)	2.738505 / 1.468490 (1.270015)	1.314897 / 4.584777 (-3.269880)	5.764442 / 3.745712 (2.018729)	3.491890 / 5.269862 (-1.777972)	2.299983 / 4.565676 (-2.265693)	0.169655 / 0.424275 (-0.254620)	0.015251 / 0.007607 (0.007643)	0.977230 / 0.226044 (0.751186)	9.697773 / 2.268929 (7.428844)	3.826928 / 55.444624 (-51.617697)	3.108238 / 6.876477 (-3.768239)	3.103242 / 2.142072 (0.961169)	1.586645 / 4.805227 (-3.218582)	0.287181 / 6.500664 (-6.213483)	0.107332 / 0.075469 (0.031863)

Benchmark: benchmark_map_filter.json

metric	filter	map fast-tokenizer batched	map identity	map identity batched	map no-op batched	map no-op batched numpy	map no-op batched pandas	map no-op batched pytorch	map no-op batched tensorflow
new / old (diff)	1.712710 / 1.841788 (-0.129077)	19.169403 / 8.074308 (11.095095)	21.777301 / 10.191392 (11.585909)	0.216918 / 0.680424 (-0.463506)	0.026551 / 0.534201 (-0.507650)	0.570383 / 0.579283 (-0.008900)	0.643885 / 0.434364 (0.209521)	0.673906 / 0.540337 (0.133568)	0.824573 / 1.386936 (-0.562363)