dataset viewer requires no-script

[severo]

No description provided.

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[github-actions]

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

Benchmark: benchmark_array_xd.json

metric	read_batch_formatted_as_numpy after write_array2d	read_batch_formatted_as_numpy after write_flattened_sequence	read_batch_formatted_as_numpy after write_nested_sequence	read_batch_unformated after write_array2d	read_batch_unformated after write_flattened_sequence	read_batch_unformated after write_nested_sequence	read_col_formatted_as_numpy after write_array2d	read_col_formatted_as_numpy after write_flattened_sequence	read_col_formatted_as_numpy after write_nested_sequence	read_col_unformated after write_array2d	read_col_unformated after write_flattened_sequence	read_col_unformated after write_nested_sequence	read_formatted_as_numpy after write_array2d	read_formatted_as_numpy after write_flattened_sequence	read_formatted_as_numpy after write_nested_sequence	read_unformated after write_array2d	read_unformated after write_flattened_sequence	read_unformated after write_nested_sequence	write_array2d	write_flattened_sequence	write_nested_sequence
new / old (diff)	0.005172 / 0.011353 (-0.006181)	0.003694 / 0.011008 (-0.007314)	0.063098 / 0.038508 (0.024590)	0.028161 / 0.023109 (0.005052)	0.262288 / 0.275898 (-0.013610)	0.281867 / 0.323480 (-0.041613)	0.004088 / 0.007986 (-0.003898)	0.002745 / 0.004328 (-0.001583)	0.049071 / 0.004250 (0.044820)	0.040629 / 0.037052 (0.003577)	0.282766 / 0.258489 (0.024277)	0.297998 / 0.293841 (0.004157)	0.028057 / 0.128546 (-0.100489)	0.010878 / 0.075646 (-0.064768)	0.207410 / 0.419271 (-0.211861)	0.035600 / 0.043533 (-0.007933)	0.260157 / 0.255139 (0.005018)	0.273252 / 0.283200 (-0.009948)	0.017403 / 0.141683 (-0.124280)	1.150798 / 1.452155 (-0.301356)	1.200485 / 1.492716 (-0.292231)

Benchmark: benchmark_getitem_100B.json

metric	get_batch_of_1024_random_rows	get_batch_of_1024_rows	get_first_row	get_last_row
new / old (diff)	0.093783 / 0.018006 (0.075777)	0.302112 / 0.000490 (0.301622)	0.000225 / 0.000200 (0.000025)	0.000043 / 0.000054 (-0.000011)

Benchmark: benchmark_indices_mapping.json

metric	select	shard	shuffle	sort	train_test_split
new / old (diff)	0.018254 / 0.037411 (-0.019158)	0.061083 / 0.014526 (0.046557)	0.074899 / 0.176557 (-0.101657)	0.119616 / 0.737135 (-0.617520)	0.075269 / 0.296338 (-0.221069)

Benchmark: benchmark_iterating.json

metric	read 5000	read 50000	read_batch 50000 10	read_batch 50000 100	read_batch 50000 1000	read_formatted numpy 5000	read_formatted pandas 5000	read_formatted tensorflow 5000	read_formatted torch 5000	read_formatted_batch numpy 5000 10	read_formatted_batch numpy 5000 1000	shuffled read 5000	shuffled read 50000	shuffled read_batch 50000 10	shuffled read_batch 50000 100	shuffled read_batch 50000 1000	shuffled read_formatted numpy 5000	shuffled read_formatted_batch numpy 5000 10	shuffled read_formatted_batch numpy 5000 1000
new / old (diff)	0.275878 / 0.215209 (0.060669)	2.694778 / 2.077655 (0.617123)	1.423810 / 1.504120 (-0.080310)	1.309444 / 1.541195 (-0.231750)	1.327898 / 1.468490 (-0.140592)	0.568621 / 4.584777 (-4.016155)	2.345849 / 3.745712 (-1.399863)	2.901281 / 5.269862 (-2.368580)	1.777959 / 4.565676 (-2.787717)	0.063539 / 0.424275 (-0.360736)	0.005011 / 0.007607 (-0.002596)	0.331212 / 0.226044 (0.105168)	3.200379 / 2.268929 (0.931451)	1.780766 / 55.444624 (-53.663859)	1.517178 / 6.876477 (-5.359299)	1.587307 / 2.142072 (-0.554765)	0.651939 / 4.805227 (-4.153288)	0.116646 / 6.500664 (-6.384018)	0.043325 / 0.075469 (-0.032144)

Benchmark: benchmark_map_filter.json

metric	filter	map fast-tokenizer batched	map identity	map identity batched	map no-op batched	map no-op batched numpy	map no-op batched pandas	map no-op batched pytorch	map no-op batched tensorflow
new / old (diff)	0.996894 / 1.841788 (-0.844894)	11.495397 / 8.074308 (3.421089)	10.255784 / 10.191392 (0.064392)	0.129006 / 0.680424 (-0.551418)	0.013967 / 0.534201 (-0.520234)	0.284847 / 0.579283 (-0.294436)	0.265610 / 0.434364 (-0.168754)	0.320176 / 0.540337 (-0.220162)	0.429526 / 1.386936 (-0.957410)

PyArrow==latest

Show updated benchmarks!

Benchmark: benchmark_array_xd.json

metric	read_batch_formatted_as_numpy after write_array2d	read_batch_formatted_as_numpy after write_flattened_sequence	read_batch_formatted_as_numpy after write_nested_sequence	read_batch_unformated after write_array2d	read_batch_unformated after write_flattened_sequence	read_batch_unformated after write_nested_sequence	read_col_formatted_as_numpy after write_array2d	read_col_formatted_as_numpy after write_flattened_sequence	read_col_formatted_as_numpy after write_nested_sequence	read_col_unformated after write_array2d	read_col_unformated after write_flattened_sequence	read_col_unformated after write_nested_sequence	read_formatted_as_numpy after write_array2d	read_formatted_as_numpy after write_flattened_sequence	read_formatted_as_numpy after write_nested_sequence	read_unformated after write_array2d	read_unformated after write_flattened_sequence	read_unformated after write_nested_sequence	write_array2d	write_flattened_sequence	write_nested_sequence
new / old (diff)	0.005582 / 0.011353 (-0.005771)	0.003867 / 0.011008 (-0.007142)	0.050416 / 0.038508 (0.011908)	0.030996 / 0.023109 (0.007887)	0.275987 / 0.275898 (0.000089)	0.289487 / 0.323480 (-0.033993)	0.005149 / 0.007986 (-0.002837)	0.002806 / 0.004328 (-0.001522)	0.049617 / 0.004250 (0.045366)	0.046949 / 0.037052 (0.009897)	0.281596 / 0.258489 (0.023107)	0.330948 / 0.293841 (0.037108)	0.049645 / 0.128546 (-0.078901)	0.010953 / 0.075646 (-0.064693)	0.058546 / 0.419271 (-0.360725)	0.034010 / 0.043533 (-0.009523)	0.270525 / 0.255139 (0.015386)	0.289749 / 0.283200 (0.006550)	0.018755 / 0.141683 (-0.122927)	1.163072 / 1.452155 (-0.289082)	1.213400 / 1.492716 (-0.279316)

Benchmark: benchmark_getitem_100B.json

metric	get_batch_of_1024_random_rows	get_batch_of_1024_rows	get_first_row	get_last_row
new / old (diff)	0.092397 / 0.018006 (0.074390)	0.299376 / 0.000490 (0.298886)	0.000211 / 0.000200 (0.000011)	0.000044 / 0.000054 (-0.000011)

Benchmark: benchmark_indices_mapping.json

metric	select	shard	shuffle	sort	train_test_split
new / old (diff)	0.022496 / 0.037411 (-0.014916)	0.076886 / 0.014526 (0.062361)	0.087186 / 0.176557 (-0.089371)	0.126092 / 0.737135 (-0.611044)	0.088832 / 0.296338 (-0.207507)

Benchmark: benchmark_iterating.json

metric	read 5000	read 50000	read_batch 50000 10	read_batch 50000 100	read_batch 50000 1000	read_formatted numpy 5000	read_formatted pandas 5000	read_formatted tensorflow 5000	read_formatted torch 5000	read_formatted_batch numpy 5000 10	read_formatted_batch numpy 5000 1000	shuffled read 5000	shuffled read 50000	shuffled read_batch 50000 10	shuffled read_batch 50000 100	shuffled read_batch 50000 1000	shuffled read_formatted numpy 5000	shuffled read_formatted_batch numpy 5000 10	shuffled read_formatted_batch numpy 5000 1000
new / old (diff)	0.288885 / 0.215209 (0.073676)	2.839851 / 2.077655 (0.762196)	1.587556 / 1.504120 (0.083436)	1.470249 / 1.541195 (-0.070945)	1.518080 / 1.468490 (0.049590)	0.569646 / 4.584777 (-4.015131)	2.417574 / 3.745712 (-1.328138)	2.737368 / 5.269862 (-2.532494)	1.784419 / 4.565676 (-2.781257)	0.064104 / 0.424275 (-0.360171)	0.005138 / 0.007607 (-0.002469)	0.346214 / 0.226044 (0.120169)	3.439541 / 2.268929 (1.170612)	1.944792 / 55.444624 (-53.499832)	1.675762 / 6.876477 (-5.200714)	1.851871 / 2.142072 (-0.290201)	0.652932 / 4.805227 (-4.152295)	0.118953 / 6.500664 (-6.381711)	0.041011 / 0.075469 (-0.034459)

Benchmark: benchmark_map_filter.json

metric	filter	map fast-tokenizer batched	map identity	map identity batched	map no-op batched	map no-op batched numpy	map no-op batched pandas	map no-op batched pytorch	map no-op batched tensorflow
new / old (diff)	1.017690 / 1.841788 (-0.824098)	12.610531 / 8.074308 (4.536223)	11.223165 / 10.191392 (1.031773)	0.131637 / 0.680424 (-0.548786)	0.016733 / 0.534201 (-0.517468)	0.288491 / 0.579283 (-0.290792)	0.275899 / 0.434364 (-0.158465)	0.331837 / 0.540337 (-0.208500)	0.421695 / 1.386936 (-0.965241)