Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

make bufr2ioda_subpfl_argo_profiles.py an executable file #795

Merged
merged 1 commit into from
Jan 4, 2024
Merged

make bufr2ioda_subpfl_argo_profiles.py an executable file #795

merged 1 commit into from
Jan 4, 2024

Conversation

RussTreadon-NOAA
Copy link
Contributor

@emilyhcliu found that permission restrictions on bufr2ioda_subpfl_argo_profiles.py prevent it from being executed. feature/chmod was created and the file permissions changed so that bufr2ioda_subpfl_argo_profiles.py is executable.

Fixes #788

@RussTreadon-NOAA RussTreadon-NOAA self-assigned this Dec 5, 2023
@ShastriPaturi
Copy link
Collaborator

Thanks @RussTreadon-NOAA

@RussTreadon-NOAA
Copy link
Contributor Author

@ShastriPaturi , would you be willing to review this PR? The only change is to change file permissions on bufr2ioda_subpfl_argo_profiles.py so that the script may be executed.

@emilyhcliu , would you mind reviewing this PR since you found this issue?

@RussTreadon-NOAA
Copy link
Contributor Author

Fixing file permissions on bufr2ioda_subpfl_argo_profiles.py is necessary, but not sufficient, for gdasprepatmiodaobs to successfully run to completion.

The job aborts with

Traceback (most recent call last):
  File "/work2/noaa/da/rtreadon/gdas-validation/global-workflow/ush/run_bufr2ioda.py", line 119, in <module>
    bufr2ioda(args.current_cycle, args.RUN, args.DMPDIR, args.config_template_dir, args.COM_OBS)
  File "/work2/noaa/da/rtreadon/gdas-validation/global-workflow/ush/python/wxflow/logger.py", line 266, in wrapper
    retval = func(*args, **kwargs)
  File "/work2/noaa/da/rtreadon/gdas-validation/global-workflow/ush/run_bufr2ioda.py", line 108, in bufr2ioda
    pool.starmap(mp_bufr_converter, zip(exename, config_files))
  File "/work2/noaa/da/python/opt/core/miniconda3/4.6.14/envs/gdasapp/lib/python3.7/multiprocessing/pool.py", line 276, in starmap
    return self._map_async(func, iterable, starmapstar, chunksize).get()
  File "/work2/noaa/da/python/opt/core/miniconda3/4.6.14/envs/gdasapp/lib/python3.7/multiprocessing/pool.py", line 657, in get
    raise self._value
wxflow.executable.ProcessError: Command exited with status 1:
'/work2/noaa/da/rtreadon/gdas-validation/global-workflow/sorc/gdas.cd/ush/ioda/bufr2ioda/bufr2ioda_subpfl_argo_profiles.py' '-c' '/work/noaa/st\
mp/rtreadon/RUNDIRS/gdas_eval_satwind_JEDI/prepatmobs.417043/subpfl_argo_profiles_2021080100.json'

I think the failure is due to the fact that my COM_OBS does not contain the dump file(s) which bufr2ioda_subpfl_argo_profiles.py processes.

This is a problem. We run gdasprepatmiodaobs as part of the GDAS-validation sprint. We can no longer successfully complete this step for other observation types due to the above failure.

Where may I find the observation dump(s) that bufr2ioda_subpfl_argo_profiles.py requires? GDAS-validation is running 2021080100.

@CoryMartin-NOAA
Copy link
Contributor

@RussTreadon-NOAA good catch, do we need separate IODA directories for atm, soca, land, aero converters?

@RussTreadon-NOAA
Copy link
Contributor Author

Unique IODA directories is an option. Figuring out the best path forward requires us to look outside GDASApp. Does g-w have separate prepioda jobs for different converters? Does / will obsproc create dump files in different directories for different components?

@ShastriPaturi
Copy link
Collaborator

@RussTreadon-NOAA the files on orion are at /work2/noaa/da/spaturi.
However I have data only for gdas.20210701 and gdas.20210702.

@CoryMartin-NOAA data for soca needs to be in a separate directory.

@CoryMartin-NOAA
Copy link
Contributor

@ShastriPaturi is that true, I thought there was only $COM_OBS in the $ROTDIR?

@ShastriPaturi
Copy link
Collaborator

@ShastriPaturi is that true, I thought there was only $COM_OBS in the $ROTDIR?

@CoryMartin-NOAA I take back.

@RussTreadon-NOAA
Copy link
Contributor Author

OK @ShastriPaturi , this is a problem. We need to move bufr2ioda_subpfl_argo_profiles.py out of ush/ioda/bufr2ioda. We can no longer successfully run the first job in GDAS-validation.

@ShastriPaturi
Copy link
Collaborator

OK @ShastriPaturi , this is a problem. We need to move bufr2ioda_subpfl_argo_profiles.py out of ush/ioda/bufr2ioda. We can no longer successfully run the first job in GDAS-validation.

Ok @RussTreadon-NOAA, till hera come back. I have data on hera that spans 202108.

@RussTreadon-NOAA
Copy link
Contributor Author

OK @ShastriPaturi , this is a problem. We need to move bufr2ioda_subpfl_argo_profiles.py out of ush/ioda/bufr2ioda. We can no longer successfully run the first job in GDAS-validation.

Ok @RussTreadon-NOAA, till hera come back. I have data on hera that spans 202108.

Adding this data to our 2021080100 GDAS-validation COM_OBS will get gdasprepatmiodaobs successfully running again. Please update this PR with the full path to the 2021080100 dump file. We can copy the file into the appropriate GDAS-validation location on Orion.

@ShastriPaturi
Copy link
Collaborator

@RussTreadon-NOAA the 2021080100 dump file on orion is at:
/work2/noaa/ng-godas/spaturi/Latency_table/insitu_BUFR/6-hourly/gdas.20210801/00/atmos/gdas.t00z.subpfl.tm00.bufr_d

@RussTreadon-NOAA
Copy link
Contributor Author

@RussTreadon-NOAA the 2021080100 dump file on orion is at: /work2/noaa/ng-godas/spaturi/Latency_table/insitu_BUFR/6-hourly/gdas.20210801/00/atmos/gdas.t00z.subpfl.tm00.bufr_d

@ShastriPaturi , I do not belong to Orion group ng-godas. Thus, I can not access /work2/noaa/ng-godas/spaturi/Latency_table/insitu_BUFR/6-hourly/gdas.20210801/00/atmos/gdas.t00z.subpfl.tm00.bufr_d.

The top-level directory /work2/noaa/ng-godas/spaturi/ has drwxr-s--- permissions.

@ShastriPaturi
Copy link
Collaborator

@RussTreadon-NOAA the 2021080100 dump file on orion is at: /work2/noaa/ng-godas/spaturi/Latency_table/insitu_BUFR/6-hourly/gdas.20210801/00/atmos/gdas.t00z.subpfl.tm00.bufr_d

@ShastriPaturi , I do not belong to Orion group ng-godas. Thus, I can not access /work2/noaa/ng-godas/spaturi/Latency_table/insitu_BUFR/6-hourly/gdas.20210801/00/atmos/gdas.t00z.subpfl.tm00.bufr_d.

The top-level directory /work2/noaa/ng-godas/spaturi/ has drwxr-s--- permissions.

@RussTreadon-NOAA sorry about that.
The file is now under:
/work2/noaa/da/spaturi/forRuss/gdas.20210801 and I have opened the permissions.

@RussTreadon-NOAA
Copy link
Contributor Author

Thank you @ShastriPaturi . I copied gdas.t00z.subpfl.tm00.bufr_d to my COMROT. gdasprepatmiodaobs failed with

2023-12-07 19:49:11,667 - INFO     - run_bufr2ioda.py: Convert subpfl_argo_profiles...
2023-12-07 19:49:11,667 - INFO     - gen_bufr2ioda_json.py: Using /work2/noaa/da/rtreadon/gdas-validation/global-workflow/sorc/gdas.cd/parm/ioda/bufr2ioda/bufr2ioda_subpfl_argo_profiles.json as input
2023-12-07 19:49:11,675 - INFO     - gen_bufr2ioda_json.py: Wrote to /work/noaa/stmp/rtreadon/RUNDIRS/gdas_eval_satwind_JEDI/prepatmobs.226229/subpfl_argo_profiles_2021080100.json
..
2023-12-07 19:49:12,160 - INFO     - run_bufr2ioda.py: Executing /work2/noaa/da/rtreadon/gdas-validation/global-workflow/sorc/gdas.cd/ush/ioda/bufr2ioda/bufr2ioda_subpfl_argo_profiles.py -c /work/noaa/stmp/rtreadon/RUNDIRS/gdas_eval_satwind_JEDI/prepatmobs.226229/subpfl_argo_profiles_2021080100.json
...
Traceback (most recent call last):
  File "/work2/noaa/da/rtreadon/gdas-validation/global-workflow/sorc/gdas.cd/ush/ioda/bufr2ioda/bufr2ioda_subpfl_argo_profiles.py", line 316, in <module>
    with open(config, "r") as json_file:
NameError: name 'config' is not defined
...
Traceback (most recent call last):
  File "/work2/noaa/da/rtreadon/gdas-validation/global-workflow/ush/run_bufr2ioda.py", line 119, in <module>
    bufr2ioda(args.current_cycle, args.RUN, args.DMPDIR, args.config_template_dir, args.COM_OBS)
  File "/work2/noaa/da/rtreadon/gdas-validation/global-workflow/ush/python/wxflow/logger.py", line 266, in wrapper
    retval = func(*args, **kwargs)
  File "/work2/noaa/da/rtreadon/gdas-validation/global-workflow/ush/run_bufr2ioda.py", line 108, in bufr2ioda
    pool.starmap(mp_bufr_converter, zip(exename, config_files))
  File "/work2/noaa/da/python/opt/core/miniconda3/4.6.14/envs/gdasapp/lib/python3.7/multiprocessing/pool.py", line 276, in starmap
    return self._map_async(func, iterable, starmapstar, chunksize).get()
  File "/work2/noaa/da/python/opt/core/miniconda3/4.6.14/envs/gdasapp/lib/python3.7/multiprocessing/pool.py", line 657, in get
    raise self._value
wxflow.executable.ProcessError: Command exited with status 1:
'/work2/noaa/da/rtreadon/gdas-validation/global-workflow/sorc/gdas.cd/ush/ioda/bufr2ioda/bufr2ioda_subpfl_argo_profiles.py' '-c' '/work/noaa/stmp/rtreadon/RUNDIRS/gdas_eval_sa\
twind_JEDI/prepatmobs.226229/subpfl_argo_profiles_2021080100.json'

@RussTreadon-NOAA
Copy link
Contributor Author

@ShastriPaturi , how do you run bufr2ioda_subpfl_argo_profiles.py?

@RussTreadon-NOAA
Copy link
Contributor Author

There is a bug in bufr2ioda_subpfl_argo_profiles.py.

    with open(config, "r") as json_file:
        config = json.load(json_file)

should be

    with open(args.config, "r") as json_file:
        config = json.load(json_file)

A rerun of prepatmiodaobs is in the queue.

@RussTreadon-NOAA
Copy link
Contributor Author

The rerun failed because the script can't find the input bufr dump file in the expected location

forrtl: Permission denied
forrtl: severe (9): permission to access file denied, unit 12, file /work/noaa/rstprod/dump/gdas.20210801/00/atmos/gdas.t00z.subpfl.tm00.bufr_d

Local file subpfl_argo_profiles_2021080100.json specifies "dump_directory": "/work/noaa/rstprod/dump",. This directory does not contain argo dump files.

The current set up uses the same variable DMPDIR for atmospheric and marine obs. We could define new g-w variables DMPDIR_ATM and DMPDIR_OCN, but I don't think this is the correct solution.

Job prepatmiodaobs is for atmospheric obs. It does not need marine obs. Likewise, job prepoceanobs is for marine obs. It does not need atmospheric obs. Each job uses DMPDIR with the variable set to different values. I'm currently trying to process marine and atmospheric obs in the same job. This doesn't work.

@RussTreadon-NOAA
Copy link
Contributor Author

This PR can not move forward until we discuss & determine how we want to run IODA observation processing for various components (atmosphere, marine, aersol, land). The current setup winds up with marine observation (subpfl_argo_profiles) as being processed by prepatmiodaobs.

@ShastriPaturi
Copy link
Collaborator

@ShastriPaturi , how do you run bufr2ioda_subpfl_argo_profiles.py?

I ran outside of global-workflow.

@ShastriPaturi
Copy link
Collaborator

There is a bug in bufr2ioda_subpfl_argo_profiles.py.

    with open(config, "r") as json_file:
        config = json.load(json_file)

should be

    with open(args.config, "r") as json_file:
        config = json.load(json_file)

A rerun of prepatmiodaobs is in the queue.

I will issue the bugfix. Thanks @RussTreadon-NOAA

@RussTreadon-NOAA
Copy link
Contributor Author

@ShastriPaturi , how do you run bufr2ioda_subpfl_argo_profiles.py?

I ran outside of global-workflow.

Thank you @ShastriPaturi for confirming that you ran outside g-w.

@guillaumevernieres guillaumevernieres merged commit bd57012 into develop Jan 4, 2024
5 checks passed
@CoryMartin-NOAA CoryMartin-NOAA deleted the feature/chmod branch March 28, 2024 15:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ShastriPaturi ShastriPaturi ShastriPaturi approved these changes

@CoryMartin-NOAA CoryMartin-NOAA CoryMartin-NOAA approved these changes

Assignees

@RussTreadon-NOAA RussTreadon-NOAA

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Permission issue for bufr2ioda_subpfl_argo_profils.py
4 participants
@RussTreadon-NOAA @ShastriPaturi @CoryMartin-NOAA @guillaumevernieres