Improve push_dataset_to_hub API + Add unit tests

[thomwolf]

What this does

• Improve push_dataset_to_hub.py API
  • Instead of --data-dir data and implicit location for raw directory, uses explicit --raw-dir data/pusht_raw with full path to directory
  • instead of --save-to-disk 1 and implicit location for output directory, uses explicit --local-dir data/pusht with full path to directory
  • Instead of --community-id lerobot and --dataset-id pusht, uses simpler --repo-id lerobot/pusht
  • Instead of --debug 1 to run on first episode, uses more configurable --episodes 0
  • Instead of --save-tests-to-disk 1 and implicit location for dataset test directory, uses explicit --tests-data-dir tests/data/pusht
  • Instead of implicit --dry-run 1, uses explicit --push-to-hub 0 (by default to 1)
  • Instead of confusing --revision, hardcode CODEBASE_VERSION
  • Adds --force-override 1 (by default 0), to not delete local directory by mistake
• When no --local-dir is provided, uses hardcoded /tmp/{REPO_ID} to cache images if needed
• Adds generic tests for push_dataset_to_hub.py
• Simplifies raw datasets download
  • Pushed all raw datasets to hub in cadene page
  • Instead of using one raw download per raw format, uses download_hub for all
• Fix a bug in xarm raw datasets that could generate push datasets instead of lift (see below)

- BREAKING CHANGE: sort the hdf5 files for aloha datasets, so episode 0 is not the same

How to try the code

# Before
python lerobot/scripts/push_dataset_to_hub.py \
--data-dir data \
--raw-format pusht_zarr \
--dataset-id pusht \
--repo-id lerobot/pusht \
--dry-run 1

# After
python lerobot/scripts/push_dataset_to_hub.py \
--raw-dir data/pusht_raw \
--raw-format pusht_zarr \
--repo-id lerobot/pusht \
--push-to-hub 0 \
--local-dir data/pusht

# Generic tests
pytest -sx tests/test_push_dataset_to_hub.py

# Backward compatibility tests
DATA_DIR=tests/data python -m pytest --run-skipped tests/test_push_dataset_to_hub.py::test_push_dataset_to_hub_pusht_backward_compatibility

How it was tested

CI

$ DATA_DIR=tests/data python -m pytest tests/test_push_dataset_to_hub.py::test_push_dataset_to_hub_pusht_zarr_format_backward_compatibility
=============================================================================== test session starts ===============================================================================
platform linux -- Python 3.10.12, pytest-8.2.1, pluggy-1.5.0
rootdir: /admin/home/remi_cadene/code/lerobot
configfile: pyproject.toml
plugins: hydra-core-1.3.2, cov-5.0.0
collected 1 item

Testing with DEVICE='cuda'

tests/test_push_dataset_to_hub.py .                                                                                                                                         [100%]

---

This change is 

[Cadene]

Nice, I prefer raw_dir and out_dir.
We will need to propagate the change to the README and other parts of the code.

Let's do it once we converged on the API for push_dataset_to_hub.
For instance, I am still wondering if we should rename it convert_to_lerobot_dataset.py with an option --push-to-hub.
With the current state of this PR, we currently have push_dataset_to_hub.py with an option --out-dir and --dry-run 1 to save locally without pushing to hub.

[thomwolf]

Updated it to fix the readme and doc. ok to merge @Cadene so I can push a next fix?

[thomwolf]

Quick updated behavior: default "save-tests-to-disk" to 0

[Cadene]

Thanks for the PR :)
We should definitly improve push_dataset_to_hub.py.

I just want to highlight that we dont have proper unit tests for push_dataset_to_hub.py for now. We prioritized other features.

If we start iterating heavily on push_dataset_to_hub.py, we might need to implement unit tests in a future PR to ensure we dont break things.

[thomwolf]

all good to merge

[thomwolf]

lgtm

[aliberts]

Left comments mainly about the interface.
I didn't look into each _format.py script, tell me if you want another pair of eyes on those.

[aliberts]

Could be nice to add a comment above each of these commands explaining in plain English what these do, kind of like what we have in here or here and maybe use some of the options to illustrate more cases (like --push-to-hub 0 or --local-dir)

[aliberts]

Thank you @Cadene & @thomwolf for this!