Fix real-world configs by adding latency

[Cadene]

What this does

• Adds latency in config, set to 0 for simulation environments and 1/fps for real-world environment

How it was tested

• For Sim, latency is 0 so no change is expected
• For real-world, we tested on Aloha2 hardware

How to checkout & try? (for the reviewer)

python lerobot/scripts/train.py \
policy=act_real \
env=dora_aloha_real \
dataset_repo_id=haixuantao/aloha_push_duplo_dora \
hydra.run.dir=/tmp/$USER/outputs/train/2024_06_01_act_aloha_push_duplo_dora \
hydra.job.name=2024_06_01_act_aloha_push_duplo_dora \
wandb.disable_artifact=true \
wandb.enable=true

---

This change is 

[alexander-soare]

Okay so, based on what we've discussed offline. I'm not sure about this.

1. My top-level feedback is we shouldn't include latency as a separate variable.

Why?

• Based on our discussion: we might not even want to account for the latency (eg: using chunking). By putting this variable here, you are making a strong suggestion to the user about how they should use delta_timestamps. To do chunking (in the way that I've suggested: starting the chunk from the current timestep or even earlier) they would have to "lie" about the latency, or change delta timestamps to not use it (in which case you have a free parameter doing nothing). This is not a far fetched scenario at all. I would like to try it soon!
• (minor) From a pure coding design perspective, it's used in 1 place yet you've assigned it far away from that place. In general, if you insist on adding another variable to the namespace, it should be defined right next to where it's used.

2. Relatively minor compared to 1. The way you document this is making some assumptions that we don't need to impose on users:

• They necessarily do inference on the rising edge of a clock which is also the same clock used to execute the action.
• They use a sleep in their code! (it's not so harmful to write this, but it can be a bit confusing like "wait - who's putting a sleep where?")

---

Not to assume up-front that you agree, but for efficiency, here's what I'd suggest if you do agree. Drop latency, put it directly in the delta_timestamps, and explain what you did there (and drop it completely from sim configs).

[Cadene]

I discussed with @aliberts . We think the proposed design of this PR is good.

This latency variable should exists because:

• it is explicit
• it is used downstream by the environment
  • in real world Dora, it is used for the dynamic sleep
  • in sim Aloha, Pusht, Xarm, at some point, it could be used to simulate the latency

[alexander-soare]

Thanks @Cadene.

If it's used elsewhere, I agree it should stay in some form.

Btw can you please share a snippet of how it is used in the Dora env? That might help a lot with this conversation. Like for example, am I allowed to change this param to 2/fps or some other number? Is this a parameter or a constant?