Trouble with vcs_info.yaml

[fernandadialzira]

I am running a pi simulation on levante with a restart rate of 5 years and right after the first 5 years run, the simulation crashes with the following message

===================================================================================================
::: Executing the step:  check_vcs_info_against_last_run    (step [17/19] of the job:  prepcompute)
===================================================================================================
Traceback (most recent call last):
  File "/home/a/a270179/.local/bin/esm_runscripts", line 33, in <module>
    sys.exit(load_entry_point('esm-tools', 'console_scripts', 'esm_runscripts')())
  File "/home/a/a270179/esm_tools/src/esm_runscripts/cli.py", line 278, in main
    Setup()
  File "/home/a/a270179/esm_tools/src/esm_runscripts/sim_objects.py", line 90, in __call__
    resubmit.maybe_resubmit(self.config)
  File "/home/a/a270179/esm_tools/src/esm_runscripts/resubmit.py", line 148, in maybe_resubmit
    nextrun = resubmit_recursively(config, jobtype=jobtype)
  File "/home/a/a270179/esm_tools/src/esm_runscripts/resubmit.py", line 189, in resubmit_recursively
    resubmit_SimulationSetup(config, cluster)
  File "/home/a/a270179/esm_tools/src/esm_runscripts/resubmit.py", line 66, in resubmit_SimulationSetup
    config["general"]["experiment_over"] = cluster_obj(kill_after_submit=False)
  File "/home/a/a270179/esm_tools/src/esm_runscripts/sim_objects.py", line 90, in __call__
    resubmit.maybe_resubmit(self.config)
  File "/home/a/a270179/esm_tools/src/esm_runscripts/resubmit.py", line 164, in maybe_resubmit
    nextrun = resubmit_recursively(
  File "/home/a/a270179/esm_tools/src/esm_runscripts/resubmit.py", line 189, in resubmit_recursively
    resubmit_SimulationSetup(config, cluster)
  File "/home/a/a270179/esm_tools/src/esm_runscripts/resubmit.py", line 66, in resubmit_SimulationSetup
    config["general"]["experiment_over"] = cluster_obj(kill_after_submit=False)
  File "/home/a/a270179/esm_tools/src/esm_runscripts/sim_objects.py", line 69, in __call__
    self.prepcompute()
  File "/home/a/a270179/esm_tools/src/esm_runscripts/sim_objects.py", line 161, in prepcompute
    self.config = prepcompute.run_job(self.config)
  File "/home/a/a270179/esm_tools/src/esm_runscripts/prepcompute.py", line 32, in run_job
    config = evaluate(config, "prepcompute", "prepcompute_recipe")
  File "/home/a/a270179/esm_tools/src/esm_runscripts/helpers.py", line 69, in evaluate
    config = esm_plugin_manager.work_through_recipe(
  File "/home/a/a270179/esm_tools/src/esm_plugin_manager/esm_plugin_manager.py", line 141, in work_through_recipe
    config = getattr(submodule, workitem)(config)
  File "/home/a/a270179/esm_tools/src/esm_runscripts/prepare.py", line 755, in check_vcs_info_against_last_run
    with open(last_exp_vcs_info_file, "r") as f:
FileNotFoundError: [Errno 2] No such file or directory: '/work/ab0246/a270179/runtime/awicm3-v3.1/experiments_production//pi_tco95_core2_003/run_18500101-18541231/log//pi_tco95_core2_003_vcs_info.yaml'

I believe it is related to the #732 update (@pgierz, @mandresm). I checked whether this file was at this path, and it actually is on the log folder of the experiment, but not to the specific run.

The path to the experiment is /work/ab0246/a270179/runtime/awicm3-v3.1/experiments_production/pi_tco95_core2_003/

[pgierz]

Hi Fernanda, which version do you have? There was a fix for that: #736. Maybe you just need to update?

[fernandadialzira]

I have 6.13.5 and I actually updated it yesterday right before running this experiment, which was around 16:19.

[pgierz]

OK, that is then indeed a bug. I guess this happens only in the very first year? I will move this Discussion to an Issue instead. For the interim, you can comment out that part in the experiment recipe; @mandresm can help you with how to do that.

[fernandadialzira]

It happens after the very first restart, so it depends on one's restart rate. In this case, after 5 years.

[JanStreffing]

Is the git feature still broken with tags instead of branches then? I did not have a chance to test it.

[antoniofis10000]

Hi all.
Same issue here when running AWICM1. (In our case restart rate=3 months).
See for example /work/ab0995/from_Mistral/ab0995/a270148/esm-experiments/AWICM.

[pgierz]

Hi Antonio,

Can you please examine the run number variable on your finished yaml file? I believe it is connected but I am still trying to gather evidence...

Best
PG

[pgierz]

And to allow you to continue working right now, you can comment out the vcs check. Fernanda can show you how

[antoniofis10000]

Hi Paul.
Commenting the lines works in the meantime.
Thanks!.
Best wishes,
Antonio.