[question] All checkpoints use the same VecNormalize statistics

[cameron-chen]

Hi,

I find parameter save_freq and eval_freq enable us to save checkpoints during training:

• save_freq: this saves checkpoints rl_model_{num_timesteps}_steps.zip
• eval_freq: this saves checkpoint best_model.zip

However, we do not have normalization statistics at the moment we save them. There is only one vecnormalize.pkl saved by function save_trained_model() after training where we save checkpoint {env_id}.zip (refer to this link).

When we evaluate one of the checkpoints (rl_model_{num_timesteps}_steps.zip, best_model.zip, {env_id}.zip), we use the same normalization statistics (refer to this link). Does this affect the evaluation? Why do we not save the normalization statistics for every checkpoints?

Thank you!

[araffin]

Hello,
good point, i did that mainly to save space, but you are right, we should give the ability to save each checkpoint stats too.

The other thing is that the normalization should converge at some point and therefore using the last stats for evaluating earlier checkpoints should not affect too much the results.

[cameron-chen]

It makes a lot of sense to me. Thank you for reply.

Saving each checkpoint stats may cost space and, in general, is not necessary. Probably, It is reasonable and practical to save the stats of best_model.zip and {env_id}.zip.

p.s. the code does save checkpoint stat for best_model.zip (refer to this link) but finally the stats is replaced by the checkpoint stats for {env_id}.zip.

[araffin]

Saving each checkpoint stats may cost space and, in general, is not necessary. Probably, It is reasonable and practical to save the stats of best_model.zip and {env_id}.zip.

I would welcome a PR that does this ;)

[cameron-chen]

Hi, I actually implemented one for personal use. Let me clean it and make a PR.