Add out_dir option to eval

[alexander-soare]

What this does

Adds the ability to specify a custom output directory for the eval.py script.

Side changes:

• Change def eval to def main to avoid clash with native python eval (because it's good practice and makes it easier to read the code)
• Change the video directory to be called "video" instead of "eval" (more precise. I actually got confused for a moment and though I implemented the output directory incorrectly)

How it was tested

CI

How to checkout & try? (for the reviewer)

To check that the custom output directory works:

python lerobot/scripts/eval.py -p lerobot/diffusion_pusht --out-dir example_eval eval.n_episodes=1 eval.batch_size=1 +policy.noise_scheduler_type=DDIM policy.num_inference_steps=10

To check that the default works:

python lerobot/scripts/eval.py -p lerobot/diffusion_pusht eval.n_episodes=1 eval.batch_size=1 +policy.noise_scheduler_type=DDIM policy.num_inference_steps=10

---

This change is 

[Cadene]

Did you search in the code base for eval.py and eval and update the documentation if any? Thanks!

Also it's a bit weird that the unit tests didnt fail if the video_dir changed: video_dir=Path(out_dir) / "video"

[alexander-soare]

@Cadene yeah I did a search. eval on its own brings up a lot of hits so I did what I could. Which of the tests do you think should fail? I don't think they check for the video directory specifically.

[Cadene]

@Cadene yeah I did a search. eval on its own brings up a lot of hits so I did what I could. Which of the tests do you think should fail? I don't think they check for the video directory specifically.

I was thinking https://github.com/huggingface/lerobot/blob/main/examples/4_train_policy_with_script.md
since it write a video during evaluation in a certain directory (eval in the experiment directory).

[Cadene]

Thanks ;) Ping me for second review

[Cadene]

What do you think?

Suggested change

	video_dir=Path(out_dir) / "video",
	video_dir=Path(out_dir) / "eval" / "videos_step_{step:09d}",

[alexander-soare]

Good catch. PTAL at my changes.

[Cadene]

What do you think?

Suggested change

	video_dir=Path(out_dir) / "video",
	videos_dir=Path(out_dir) / "videos",

[alexander-soare]

Fine with that. Done.

[Cadene]

YOLO without tests

[tlpss]

FYI, I had to reinstall the package (even though it was installed in editable mode) to overcome this error:

Error executing job with overrides: ['+experiment=pusht-act']
Traceback (most recent call last):
  File "/home/tlips/Code/lerobot/lerobot/scripts/train.py", line 424, in train_cli
    train(
  File "/home/tlips/Code/lerobot/lerobot/scripts/train.py", line 414, in train
    evaluate_and_checkpoint_if_needed(step + 1)
  File "/home/tlips/Code/lerobot/lerobot/scripts/train.py", line 337, in evaluate_and_checkpoint_if_needed
    eval_info = eval_policy(
TypeError: eval_policy() got an unexpected keyword argument 'videos_dir'

Seems to work fine now though!

PS. You forgot to update the docstring here

[alexander-soare]

@tlpss thanks a lot for pointing that out! Ugh I knew I'd miss something...