Preserve split order in DataFilesDict

[albertvillanova]

After investigation, I have found that this copy forces the splits to be sorted alphabetically:

datasets/src/datasets/builder.py

Line 556 in 029227a

builder_config = copy.deepcopy(builder_config)

This PR removes the alphabetically sort of DataFilesDict keys.

• Note that for a dict, the order of keys is relevant when hashing:

  hash1 = Hasher.hash({'train': 'train.csv', 'test': 'test.csv'})
  hash2 = Hasher.hash({'test': 'test.csv', 'train': 'train.csv'})
  assert hash1 != hash2

• The DataFilesDict is a subclass of dict, thus the order should be relevant as well

  hash1 = Hasher.hash(DataFilesDict({'train': 'train.csv', 'test': 'test.csv'}))
  hash2 = Hasher.hash(DataFilesDict({'test': 'test.csv', 'train': 'train.csv'}))
  assert hash1 != hash2

Fix #6196.

[HuggingFaceDocBuilderDev]

The documentation is not available anymore as the PR was closed or merged.

[github-actions]

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

Benchmark: benchmark_array_xd.json

metric	read_batch_formatted_as_numpy after write_array2d	read_batch_formatted_as_numpy after write_flattened_sequence	read_batch_formatted_as_numpy after write_nested_sequence	read_batch_unformated after write_array2d	read_batch_unformated after write_flattened_sequence	read_batch_unformated after write_nested_sequence	read_col_formatted_as_numpy after write_array2d	read_col_formatted_as_numpy after write_flattened_sequence	read_col_formatted_as_numpy after write_nested_sequence	read_col_unformated after write_array2d	read_col_unformated after write_flattened_sequence	read_col_unformated after write_nested_sequence	read_formatted_as_numpy after write_array2d	read_formatted_as_numpy after write_flattened_sequence	read_formatted_as_numpy after write_nested_sequence	read_unformated after write_array2d	read_unformated after write_flattened_sequence	read_unformated after write_nested_sequence	write_array2d	write_flattened_sequence	write_nested_sequence
new / old (diff)	0.007621 / 0.011353 (-0.003732)	0.004534 / 0.011008 (-0.006475)	0.099834 / 0.038508 (0.061326)	0.083029 / 0.023109 (0.059920)	0.387559 / 0.275898 (0.111661)	0.422453 / 0.323480 (0.098973)	0.006070 / 0.007986 (-0.001916)	0.003725 / 0.004328 (-0.000604)	0.075923 / 0.004250 (0.071672)	0.060578 / 0.037052 (0.023525)	0.403569 / 0.258489 (0.145079)	0.444991 / 0.293841 (0.151150)	0.035847 / 0.128546 (-0.092699)	0.009872 / 0.075646 (-0.065774)	0.335506 / 0.419271 (-0.083766)	0.060509 / 0.043533 (0.016976)	0.381034 / 0.255139 (0.125895)	0.426938 / 0.283200 (0.143738)	0.027662 / 0.141683 (-0.114021)	1.729565 / 1.452155 (0.277410)	1.842082 / 1.492716 (0.349366)

Benchmark: benchmark_getitem_100B.json

metric	get_batch_of_1024_random_rows	get_batch_of_1024_rows	get_first_row	get_last_row
new / old (diff)	0.230371 / 0.018006 (0.212365)	0.518216 / 0.000490 (0.517726)	0.003897 / 0.000200 (0.003697)	0.000087 / 0.000054 (0.000033)

Benchmark: benchmark_indices_mapping.json

metric	select	shard	shuffle	sort	train_test_split
new / old (diff)	0.031942 / 0.037411 (-0.005470)	0.096609 / 0.014526 (0.082083)	0.112707 / 0.176557 (-0.063850)	0.178849 / 0.737135 (-0.558286)	0.112793 / 0.296338 (-0.183546)

Benchmark: benchmark_iterating.json

metric	read 5000	read 50000	read_batch 50000 10	read_batch 50000 100	read_batch 50000 1000	read_formatted numpy 5000	read_formatted pandas 5000	read_formatted tensorflow 5000	read_formatted torch 5000	read_formatted_batch numpy 5000 10	read_formatted_batch numpy 5000 1000	shuffled read 5000	shuffled read 50000	shuffled read_batch 50000 10	shuffled read_batch 50000 100	shuffled read_batch 50000 1000	shuffled read_formatted numpy 5000	shuffled read_formatted_batch numpy 5000 10	shuffled read_formatted_batch numpy 5000 1000
new / old (diff)	0.445896 / 0.215209 (0.230687)	4.451173 / 2.077655 (2.373519)	2.183380 / 1.504120 (0.679260)	1.991583 / 1.541195 (0.450388)	2.096219 / 1.468490 (0.627729)	0.566692 / 4.584777 (-4.018085)	4.078278 / 3.745712 (0.332566)	3.787950 / 5.269862 (-1.481911)	2.372651 / 4.565676 (-2.193025)	0.065500 / 0.424275 (-0.358775)	0.008918 / 0.007607 (0.001311)	0.535589 / 0.226044 (0.309545)	5.364130 / 2.268929 (3.095201)	2.805381 / 55.444624 (-52.639244)	2.350769 / 6.876477 (-4.525708)	2.592887 / 2.142072 (0.450814)	0.675475 / 4.805227 (-4.129752)	0.153907 / 6.500664 (-6.346757)	0.071138 / 0.075469 (-0.004331)

Benchmark: benchmark_map_filter.json

metric	filter	map fast-tokenizer batched	map identity	map identity batched	map no-op batched	map no-op batched numpy	map no-op batched pandas	map no-op batched pytorch	map no-op batched tensorflow
new / old (diff)	1.498236 / 1.841788 (-0.343552)	22.810460 / 8.074308 (14.736152)	16.275035 / 10.191392 (6.083643)	0.200242 / 0.680424 (-0.480182)	0.021553 / 0.534201 (-0.512648)	0.469437 / 0.579283 (-0.109846)	0.477752 / 0.434364 (0.043388)	0.537411 / 0.540337 (-0.002927)	0.741730 / 1.386936 (-0.645206)

PyArrow==latest

Show updated benchmarks!

Benchmark: benchmark_array_xd.json

metric	read_batch_formatted_as_numpy after write_array2d	read_batch_formatted_as_numpy after write_flattened_sequence	read_batch_formatted_as_numpy after write_nested_sequence	read_batch_unformated after write_array2d	read_batch_unformated after write_flattened_sequence	read_batch_unformated after write_nested_sequence	read_col_formatted_as_numpy after write_array2d	read_col_formatted_as_numpy after write_flattened_sequence	read_col_formatted_as_numpy after write_nested_sequence	read_col_unformated after write_array2d	read_col_unformated after write_flattened_sequence	read_col_unformated after write_nested_sequence	read_formatted_as_numpy after write_array2d	read_formatted_as_numpy after write_flattened_sequence	read_formatted_as_numpy after write_nested_sequence	read_unformated after write_array2d	read_unformated after write_flattened_sequence	read_unformated after write_nested_sequence	write_array2d	write_flattened_sequence	write_nested_sequence
new / old (diff)	0.008009 / 0.011353 (-0.003344)	0.004626 / 0.011008 (-0.006382)	0.074871 / 0.038508 (0.036363)	0.085214 / 0.023109 (0.062105)	0.478057 / 0.275898 (0.202159)	0.522038 / 0.323480 (0.198558)	0.007055 / 0.007986 (-0.000931)	0.003813 / 0.004328 (-0.000515)	0.076238 / 0.004250 (0.071988)	0.065738 / 0.037052 (0.028686)	0.484391 / 0.258489 (0.225902)	0.524425 / 0.293841 (0.230584)	0.038375 / 0.128546 (-0.090171)	0.009964 / 0.075646 (-0.065682)	0.084027 / 0.419271 (-0.335245)	0.056979 / 0.043533 (0.013447)	0.486910 / 0.255139 (0.231771)	0.501185 / 0.283200 (0.217985)	0.027000 / 0.141683 (-0.114683)	1.767378 / 1.452155 (0.315224)	1.870511 / 1.492716 (0.377795)

Benchmark: benchmark_getitem_100B.json

metric	get_batch_of_1024_random_rows	get_batch_of_1024_rows	get_first_row	get_last_row
new / old (diff)	0.267067 / 0.018006 (0.249061)	0.501714 / 0.000490 (0.501224)	0.012379 / 0.000200 (0.012179)	0.000129 / 0.000054 (0.000075)

Benchmark: benchmark_indices_mapping.json

metric	select	shard	shuffle	sort	train_test_split
new / old (diff)	0.036706 / 0.037411 (-0.000706)	0.110064 / 0.014526 (0.095538)	0.124896 / 0.176557 (-0.051660)	0.186730 / 0.737135 (-0.550405)	0.123501 / 0.296338 (-0.172837)

Benchmark: benchmark_iterating.json

metric	read 5000	read 50000	read_batch 50000 10	read_batch 50000 100	read_batch 50000 1000	read_formatted numpy 5000	read_formatted pandas 5000	read_formatted tensorflow 5000	read_formatted torch 5000	read_formatted_batch numpy 5000 10	read_formatted_batch numpy 5000 1000	shuffled read 5000	shuffled read 50000	shuffled read_batch 50000 10	shuffled read_batch 50000 100	shuffled read_batch 50000 1000	shuffled read_formatted numpy 5000	shuffled read_formatted_batch numpy 5000 10	shuffled read_formatted_batch numpy 5000 1000
new / old (diff)	0.510793 / 0.215209 (0.295583)	5.133056 / 2.077655 (3.055401)	2.776456 / 1.504120 (1.272336)	2.595557 / 1.541195 (1.054362)	2.717922 / 1.468490 (1.249432)	0.578333 / 4.584777 (-4.006444)	4.169935 / 3.745712 (0.424223)	3.800078 / 5.269862 (-1.469784)	2.385866 / 4.565676 (-2.179810)	0.068114 / 0.424275 (-0.356161)	0.008771 / 0.007607 (0.001164)	0.597894 / 0.226044 (0.371850)	5.970293 / 2.268929 (3.701364)	3.352715 / 55.444624 (-52.091909)	2.972062 / 6.876477 (-3.904415)	3.179232 / 2.142072 (1.037160)	0.689838 / 4.805227 (-4.115389)	0.154890 / 6.500664 (-6.345774)	0.072321 / 0.075469 (-0.003148)

Benchmark: benchmark_map_filter.json

metric	filter	map fast-tokenizer batched	map identity	map identity batched	map no-op batched	map no-op batched numpy	map no-op batched pandas	map no-op batched pytorch	map no-op batched tensorflow
new / old (diff)	1.613666 / 1.841788 (-0.228121)	23.441538 / 8.074308 (15.367230)	17.105417 / 10.191392 (6.914025)	0.171449 / 0.680424 (-0.508975)	0.023257 / 0.534201 (-0.510944)	0.466724 / 0.579283 (-0.112559)	0.470835 / 0.434364 (0.036471)	0.561860 / 0.540337 (0.021523)	0.759048 / 1.386936 (-0.627888)

[github-actions]

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

Benchmark: benchmark_array_xd.json

metric	read_batch_formatted_as_numpy after write_array2d	read_batch_formatted_as_numpy after write_flattened_sequence	read_batch_formatted_as_numpy after write_nested_sequence	read_batch_unformated after write_array2d	read_batch_unformated after write_flattened_sequence	read_batch_unformated after write_nested_sequence	read_col_formatted_as_numpy after write_array2d	read_col_formatted_as_numpy after write_flattened_sequence	read_col_formatted_as_numpy after write_nested_sequence	read_col_unformated after write_array2d	read_col_unformated after write_flattened_sequence	read_col_unformated after write_nested_sequence	read_formatted_as_numpy after write_array2d	read_formatted_as_numpy after write_flattened_sequence	read_formatted_as_numpy after write_nested_sequence	read_unformated after write_array2d	read_unformated after write_flattened_sequence	read_unformated after write_nested_sequence	write_array2d	write_flattened_sequence	write_nested_sequence
new / old (diff)	0.007557 / 0.011353 (-0.003796)	0.004211 / 0.011008 (-0.006797)	0.096243 / 0.038508 (0.057735)	0.083603 / 0.023109 (0.060493)	0.367114 / 0.275898 (0.091216)	0.415182 / 0.323480 (0.091702)	0.005796 / 0.007986 (-0.002189)	0.003791 / 0.004328 (-0.000537)	0.073505 / 0.004250 (0.069254)	0.060335 / 0.037052 (0.023283)	0.392182 / 0.258489 (0.133693)	0.421315 / 0.293841 (0.127474)	0.036128 / 0.128546 (-0.092419)	0.009953 / 0.075646 (-0.065693)	0.338965 / 0.419271 (-0.080307)	0.061006 / 0.043533 (0.017473)	0.372317 / 0.255139 (0.117178)	0.414367 / 0.283200 (0.131167)	0.026970 / 0.141683 (-0.114713)	1.730381 / 1.452155 (0.278227)	1.808340 / 1.492716 (0.315624)

Benchmark: benchmark_getitem_100B.json

metric	get_batch_of_1024_random_rows	get_batch_of_1024_rows	get_first_row	get_last_row
new / old (diff)	0.222622 / 0.018006 (0.204615)	0.474064 / 0.000490 (0.473574)	0.004817 / 0.000200 (0.004617)	0.000089 / 0.000054 (0.000034)

Benchmark: benchmark_indices_mapping.json

metric	select	shard	shuffle	sort	train_test_split
new / old (diff)	0.032528 / 0.037411 (-0.004883)	0.097457 / 0.014526 (0.082931)	0.112273 / 0.176557 (-0.064283)	0.177953 / 0.737135 (-0.559182)	0.112358 / 0.296338 (-0.183981)

Benchmark: benchmark_iterating.json

metric	read 5000	read 50000	read_batch 50000 10	read_batch 50000 100	read_batch 50000 1000	read_formatted numpy 5000	read_formatted pandas 5000	read_formatted tensorflow 5000	read_formatted torch 5000	read_formatted_batch numpy 5000 10	read_formatted_batch numpy 5000 1000	shuffled read 5000	shuffled read 50000	shuffled read_batch 50000 10	shuffled read_batch 50000 100	shuffled read_batch 50000 1000	shuffled read_formatted numpy 5000	shuffled read_formatted_batch numpy 5000 10	shuffled read_formatted_batch numpy 5000 1000
new / old (diff)	0.442601 / 0.215209 (0.227392)	4.442065 / 2.077655 (2.364410)	2.156813 / 1.504120 (0.652694)	1.970289 / 1.541195 (0.429094)	2.052878 / 1.468490 (0.584388)	0.562661 / 4.584777 (-4.022116)	4.255529 / 3.745712 (0.509817)	3.767650 / 5.269862 (-1.502212)	2.431078 / 4.565676 (-2.134598)	0.065624 / 0.424275 (-0.358651)	0.008738 / 0.007607 (0.001131)	0.546839 / 0.226044 (0.320795)	5.362863 / 2.268929 (3.093934)	2.695924 / 55.444624 (-52.748701)	2.334589 / 6.876477 (-4.541888)	2.530757 / 2.142072 (0.388685)	0.675991 / 4.805227 (-4.129236)	0.153852 / 6.500664 (-6.346813)	0.069189 / 0.075469 (-0.006280)

Benchmark: benchmark_map_filter.json

metric	filter	map fast-tokenizer batched	map identity	map identity batched	map no-op batched	map no-op batched numpy	map no-op batched pandas	map no-op batched pytorch	map no-op batched tensorflow
new / old (diff)	1.522916 / 1.841788 (-0.318872)	21.515907 / 8.074308 (13.441599)	16.411708 / 10.191392 (6.220316)	0.168245 / 0.680424 (-0.512179)	0.021165 / 0.534201 (-0.513036)	0.461838 / 0.579283 (-0.117446)	0.488867 / 0.434364 (0.054503)	0.536278 / 0.540337 (-0.004059)	0.766690 / 1.386936 (-0.620246)

PyArrow==latest

Show updated benchmarks!

Benchmark: benchmark_array_xd.json

metric	read_batch_formatted_as_numpy after write_array2d	read_batch_formatted_as_numpy after write_flattened_sequence	read_batch_formatted_as_numpy after write_nested_sequence	read_batch_unformated after write_array2d	read_batch_unformated after write_flattened_sequence	read_batch_unformated after write_nested_sequence	read_col_formatted_as_numpy after write_array2d	read_col_formatted_as_numpy after write_flattened_sequence	read_col_formatted_as_numpy after write_nested_sequence	read_col_unformated after write_array2d	read_col_unformated after write_flattened_sequence	read_col_unformated after write_nested_sequence	read_formatted_as_numpy after write_array2d	read_formatted_as_numpy after write_flattened_sequence	read_formatted_as_numpy after write_nested_sequence	read_unformated after write_array2d	read_unformated after write_flattened_sequence	read_unformated after write_nested_sequence	write_array2d	write_flattened_sequence	write_nested_sequence
new / old (diff)	0.007683 / 0.011353 (-0.003670)	0.004401 / 0.011008 (-0.006608)	0.075463 / 0.038508 (0.036955)	0.081737 / 0.023109 (0.058628)	0.466469 / 0.275898 (0.190571)	0.514909 / 0.323480 (0.191429)	0.006106 / 0.007986 (-0.001880)	0.003936 / 0.004328 (-0.000393)	0.076773 / 0.004250 (0.072523)	0.061025 / 0.037052 (0.023973)	0.473348 / 0.258489 (0.214858)	0.525326 / 0.293841 (0.231485)	0.038224 / 0.128546 (-0.090322)	0.009559 / 0.075646 (-0.066087)	0.080847 / 0.419271 (-0.338424)	0.056738 / 0.043533 (0.013205)	0.475116 / 0.255139 (0.219977)	0.494689 / 0.283200 (0.211490)	0.029364 / 0.141683 (-0.112319)	1.796681 / 1.452155 (0.344527)	1.850600 / 1.492716 (0.357884)

Benchmark: benchmark_getitem_100B.json

metric	get_batch_of_1024_random_rows	get_batch_of_1024_rows	get_first_row	get_last_row
new / old (diff)	0.327126 / 0.018006 (0.309119)	0.469186 / 0.000490 (0.468696)	0.050600 / 0.000200 (0.050400)	0.000439 / 0.000054 (0.000385)

Benchmark: benchmark_indices_mapping.json

metric	select	shard	shuffle	sort	train_test_split
new / old (diff)	0.036710 / 0.037411 (-0.000701)	0.108669 / 0.014526 (0.094143)	0.119808 / 0.176557 (-0.056748)	0.181501 / 0.737135 (-0.555634)	0.121487 / 0.296338 (-0.174852)

Benchmark: benchmark_iterating.json

metric	read 5000	read 50000	read_batch 50000 10	read_batch 50000 100	read_batch 50000 1000	read_formatted numpy 5000	read_formatted pandas 5000	read_formatted tensorflow 5000	read_formatted torch 5000	read_formatted_batch numpy 5000 10	read_formatted_batch numpy 5000 1000	shuffled read 5000	shuffled read 50000	shuffled read_batch 50000 10	shuffled read_batch 50000 100	shuffled read_batch 50000 1000	shuffled read_formatted numpy 5000	shuffled read_formatted_batch numpy 5000 10	shuffled read_formatted_batch numpy 5000 1000
new / old (diff)	0.509076 / 0.215209 (0.293867)	5.056970 / 2.077655 (2.979316)	2.775958 / 1.504120 (1.271838)	2.592548 / 1.541195 (1.051353)	2.654381 / 1.468490 (1.185890)	0.557407 / 4.584777 (-4.027370)	4.418232 / 3.745712 (0.672519)	3.698072 / 5.269862 (-1.571790)	2.380607 / 4.565676 (-2.185069)	0.066242 / 0.424275 (-0.358034)	0.008350 / 0.007607 (0.000743)	0.572354 / 0.226044 (0.346309)	5.857637 / 2.268929 (3.588709)	3.242512 / 55.444624 (-52.202112)	2.891144 / 6.876477 (-3.985332)	3.217987 / 2.142072 (1.075915)	0.676049 / 4.805227 (-4.129178)	0.155515 / 6.500664 (-6.345149)	0.068616 / 0.075469 (-0.006853)

Benchmark: benchmark_map_filter.json

metric	filter	map fast-tokenizer batched	map identity	map identity batched	map no-op batched	map no-op batched numpy	map no-op batched pandas	map no-op batched pytorch	map no-op batched tensorflow
new / old (diff)	1.670048 / 1.841788 (-0.171740)	22.629573 / 8.074308 (14.555265)	16.887676 / 10.191392 (6.696284)	0.168571 / 0.680424 (-0.511853)	0.023361 / 0.534201 (-0.510840)	0.463358 / 0.579283 (-0.115925)	0.463278 / 0.434364 (0.028914)	0.602397 / 0.540337 (0.062060)	0.793249 / 1.386936 (-0.593687)

[lhoestq]

good catch !

[github-actions]

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

Benchmark: benchmark_array_xd.json

metric	read_batch_formatted_as_numpy after write_array2d	read_batch_formatted_as_numpy after write_flattened_sequence	read_batch_formatted_as_numpy after write_nested_sequence	read_batch_unformated after write_array2d	read_batch_unformated after write_flattened_sequence	read_batch_unformated after write_nested_sequence	read_col_formatted_as_numpy after write_array2d	read_col_formatted_as_numpy after write_flattened_sequence	read_col_formatted_as_numpy after write_nested_sequence	read_col_unformated after write_array2d	read_col_unformated after write_flattened_sequence	read_col_unformated after write_nested_sequence	read_formatted_as_numpy after write_array2d	read_formatted_as_numpy after write_flattened_sequence	read_formatted_as_numpy after write_nested_sequence	read_unformated after write_array2d	read_unformated after write_flattened_sequence	read_unformated after write_nested_sequence	write_array2d	write_flattened_sequence	write_nested_sequence
new / old (diff)	0.006693 / 0.011353 (-0.004660)	0.004100 / 0.011008 (-0.006908)	0.084166 / 0.038508 (0.045658)	0.074469 / 0.023109 (0.051360)	0.356092 / 0.275898 (0.080194)	0.392389 / 0.323480 (0.068909)	0.003996 / 0.007986 (-0.003990)	0.004020 / 0.004328 (-0.000308)	0.064997 / 0.004250 (0.060747)	0.053897 / 0.037052 (0.016845)	0.362942 / 0.258489 (0.104453)	0.408694 / 0.293841 (0.114854)	0.031656 / 0.128546 (-0.096890)	0.008713 / 0.075646 (-0.066933)	0.289306 / 0.419271 (-0.129966)	0.053067 / 0.043533 (0.009534)	0.358740 / 0.255139 (0.103601)	0.393347 / 0.283200 (0.110147)	0.025430 / 0.141683 (-0.116253)	1.486114 / 1.452155 (0.033959)	1.572698 / 1.492716 (0.079981)

Benchmark: benchmark_getitem_100B.json

metric	get_batch_of_1024_random_rows	get_batch_of_1024_rows	get_first_row	get_last_row
new / old (diff)	0.215423 / 0.018006 (0.197417)	0.467694 / 0.000490 (0.467204)	0.003965 / 0.000200 (0.003765)	0.000112 / 0.000054 (0.000057)

Benchmark: benchmark_indices_mapping.json

metric	select	shard	shuffle	sort	train_test_split
new / old (diff)	0.027936 / 0.037411 (-0.009475)	0.084235 / 0.014526 (0.069709)	0.136275 / 0.176557 (-0.040282)	0.151154 / 0.737135 (-0.585982)	0.185592 / 0.296338 (-0.110747)

Benchmark: benchmark_iterating.json

metric	read 5000	read 50000	read_batch 50000 10	read_batch 50000 100	read_batch 50000 1000	read_formatted numpy 5000	read_formatted pandas 5000	read_formatted tensorflow 5000	read_formatted torch 5000	read_formatted_batch numpy 5000 10	read_formatted_batch numpy 5000 1000	shuffled read 5000	shuffled read 50000	shuffled read_batch 50000 10	shuffled read_batch 50000 100	shuffled read_batch 50000 1000	shuffled read_formatted numpy 5000	shuffled read_formatted_batch numpy 5000 10	shuffled read_formatted_batch numpy 5000 1000
new / old (diff)	0.393784 / 0.215209 (0.178575)	3.927878 / 2.077655 (1.850223)	1.961216 / 1.504120 (0.457096)	1.802264 / 1.541195 (0.261069)	1.971186 / 1.468490 (0.502696)	0.487981 / 4.584777 (-4.096796)	3.649046 / 3.745712 (-0.096666)	3.302471 / 5.269862 (-1.967391)	2.058075 / 4.565676 (-2.507602)	0.057072 / 0.424275 (-0.367203)	0.007624 / 0.007607 (0.000017)	0.470139 / 0.226044 (0.244095)	4.697711 / 2.268929 (2.428783)	2.494813 / 55.444624 (-52.949811)	2.133084 / 6.876477 (-4.743393)	2.329740 / 2.142072 (0.187667)	0.585857 / 4.805227 (-4.219371)	0.134442 / 6.500664 (-6.366223)	0.060860 / 0.075469 (-0.014609)

Benchmark: benchmark_map_filter.json

metric	filter	map fast-tokenizer batched	map identity	map identity batched	map no-op batched	map no-op batched numpy	map no-op batched pandas	map no-op batched pytorch	map no-op batched tensorflow
new / old (diff)	1.248504 / 1.841788 (-0.593283)	19.448427 / 8.074308 (11.374119)	14.446139 / 10.191392 (4.254747)	0.168081 / 0.680424 (-0.512342)	0.018028 / 0.534201 (-0.516173)	0.395061 / 0.579283 (-0.184222)	0.418777 / 0.434364 (-0.015587)	0.454509 / 0.540337 (-0.085828)	0.628488 / 1.386936 (-0.758448)

PyArrow==latest

Show updated benchmarks!

Benchmark: benchmark_array_xd.json

metric	read_batch_formatted_as_numpy after write_array2d	read_batch_formatted_as_numpy after write_flattened_sequence	read_batch_formatted_as_numpy after write_nested_sequence	read_batch_unformated after write_array2d	read_batch_unformated after write_flattened_sequence	read_batch_unformated after write_nested_sequence	read_col_formatted_as_numpy after write_array2d	read_col_formatted_as_numpy after write_flattened_sequence	read_col_formatted_as_numpy after write_nested_sequence	read_col_unformated after write_array2d	read_col_unformated after write_flattened_sequence	read_col_unformated after write_nested_sequence	read_formatted_as_numpy after write_array2d	read_formatted_as_numpy after write_flattened_sequence	read_formatted_as_numpy after write_nested_sequence	read_unformated after write_array2d	read_unformated after write_flattened_sequence	read_unformated after write_nested_sequence	write_array2d	write_flattened_sequence	write_nested_sequence
new / old (diff)	0.006946 / 0.011353 (-0.004406)	0.004096 / 0.011008 (-0.006912)	0.065322 / 0.038508 (0.026813)	0.074336 / 0.023109 (0.051227)	0.405327 / 0.275898 (0.129429)	0.436878 / 0.323480 (0.113398)	0.006083 / 0.007986 (-0.001902)	0.003345 / 0.004328 (-0.000984)	0.065725 / 0.004250 (0.061474)	0.056398 / 0.037052 (0.019345)	0.406906 / 0.258489 (0.148417)	0.443330 / 0.293841 (0.149489)	0.033036 / 0.128546 (-0.095510)	0.008503 / 0.075646 (-0.067144)	0.071865 / 0.419271 (-0.347406)	0.048956 / 0.043533 (0.005423)	0.404579 / 0.255139 (0.149440)	0.424904 / 0.283200 (0.141704)	0.021786 / 0.141683 (-0.119897)	1.491868 / 1.452155 (0.039713)	1.565252 / 1.492716 (0.072536)

Benchmark: benchmark_getitem_100B.json

metric	get_batch_of_1024_random_rows	get_batch_of_1024_rows	get_first_row	get_last_row
new / old (diff)	0.231363 / 0.018006 (0.213357)	0.454962 / 0.000490 (0.454472)	0.004680 / 0.000200 (0.004480)	0.000100 / 0.000054 (0.000045)

Benchmark: benchmark_indices_mapping.json

metric	select	shard	shuffle	sort	train_test_split
new / old (diff)	0.032569 / 0.037411 (-0.004843)	0.094928 / 0.014526 (0.080402)	0.108096 / 0.176557 (-0.068461)	0.158727 / 0.737135 (-0.578409)	0.106951 / 0.296338 (-0.189387)

Benchmark: benchmark_iterating.json

metric	read 5000	read 50000	read_batch 50000 10	read_batch 50000 100	read_batch 50000 1000	read_formatted numpy 5000	read_formatted pandas 5000	read_formatted tensorflow 5000	read_formatted torch 5000	read_formatted_batch numpy 5000 10	read_formatted_batch numpy 5000 1000	shuffled read 5000	shuffled read 50000	shuffled read_batch 50000 10	shuffled read_batch 50000 100	shuffled read_batch 50000 1000	shuffled read_formatted numpy 5000	shuffled read_formatted_batch numpy 5000 10	shuffled read_formatted_batch numpy 5000 1000
new / old (diff)	0.431469 / 0.215209 (0.216260)	4.283929 / 2.077655 (2.206274)	2.283891 / 1.504120 (0.779771)	2.118172 / 1.541195 (0.576977)	2.192628 / 1.468490 (0.724138)	0.492026 / 4.584777 (-4.092751)	3.692126 / 3.745712 (-0.053587)	3.269827 / 5.269862 (-2.000035)	2.028948 / 4.565676 (-2.536728)	0.057932 / 0.424275 (-0.366344)	0.007301 / 0.007607 (-0.000306)	0.508411 / 0.226044 (0.282367)	5.072803 / 2.268929 (2.803875)	2.756532 / 55.444624 (-52.688092)	2.432192 / 6.876477 (-4.444285)	2.654864 / 2.142072 (0.512791)	0.589458 / 4.805227 (-4.215769)	0.133924 / 6.500664 (-6.366740)	0.060764 / 0.075469 (-0.014705)

Benchmark: benchmark_map_filter.json

metric	filter	map fast-tokenizer batched	map identity	map identity batched	map no-op batched	map no-op batched numpy	map no-op batched pandas	map no-op batched pytorch	map no-op batched tensorflow
new / old (diff)	1.350737 / 1.841788 (-0.491051)	20.265217 / 8.074308 (12.190909)	14.969039 / 10.191392 (4.777647)	0.164226 / 0.680424 (-0.516198)	0.020090 / 0.534201 (-0.514111)	0.397010 / 0.579283 (-0.182273)	0.412927 / 0.434364 (-0.021437)	0.473931 / 0.540337 (-0.066406)	0.653462 / 1.386936 (-0.733474)