-
Notifications
You must be signed in to change notification settings - Fork 641
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add out_dir option to eval #244
Conversation
Did you search in the code base for Also it's a bit weird that the unit tests didnt fail if the |
@Cadene yeah I did a search. |
I was thinking https://github.com/huggingface/lerobot/blob/main/examples/4_train_policy_with_script.md |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks ;) Ping me for second review
lerobot/scripts/train.py
Outdated
@@ -332,7 +332,7 @@ def evaluate_and_checkpoint_if_needed(step): | |||
eval_env, | |||
policy, | |||
cfg.eval.n_episodes, | |||
video_dir=Path(out_dir) / "eval", | |||
video_dir=Path(out_dir) / "video", |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What do you think?
video_dir=Path(out_dir) / "video", | |
video_dir=Path(out_dir) / "eval" / "videos_step_{step:09d}", |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Good catch. PTAL at my changes.
lerobot/scripts/eval.py
Outdated
@@ -546,7 +543,7 @@ def eval( | |||
policy, | |||
hydra_cfg.eval.n_episodes, | |||
max_episodes_rendered=10, | |||
video_dir=Path(out_dir) / "eval", | |||
video_dir=Path(out_dir) / "video", |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What do you think?
video_dir=Path(out_dir) / "video", | |
videos_dir=Path(out_dir) / "videos", |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fine with that. Done.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
YOLO without tests
FYI, I had to reinstall the package (even though it was installed in editable mode) to overcome this error:
Seems to work fine now though! PS. You forgot to update the docstring here |
@tlpss thanks a lot for pointing that out! Ugh I knew I'd miss something... |
What this does
Adds the ability to specify a custom output directory for the
eval.py
script.Side changes:
def eval
todef main
to avoid clash with native pythoneval
(because it's good practice and makes it easier to read the code)How it was tested
CI
How to checkout & try? (for the reviewer)
To check that the custom output directory works:
python lerobot/scripts/eval.py -p lerobot/diffusion_pusht --out-dir example_eval eval.n_episodes=1 eval.batch_size=1 +policy.noise_scheduler_type=DDIM policy.num_inference_steps=10
To check that the default works:
python lerobot/scripts/eval.py -p lerobot/diffusion_pusht eval.n_episodes=1 eval.batch_size=1 +policy.noise_scheduler_type=DDIM policy.num_inference_steps=10
This change is