-
Notifications
You must be signed in to change notification settings - Fork 108
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Test new BNU soil type and VIIRS vegetation type data #821
Comments
Sanath is testing this dataset. However, he does not need the sand and clay records. I will create a branch where |
Test the new data with the Using f9e63a3, I tried creating a global C1152 grid using the "fraction of each category" method (
After 30 minutes, the job timed out before making it past the vegetation type processing, which is the first field processed. I reran the same grid, but with
The job ran successfully in about 6 minutes. Using |
Run some timing tests using the For a C384 grid:
It ran successfully in 22 minutes. |
The above test was rerun at C768. It took 25 minutes. (Note: The first try timed out after 30 minutes. The veg file was created at about 28 minutes. So, I assumed this test would take about an hour. That was not the case this time. Don't know why run times are so variable on Hera. Requesting 'exclusive' use of the nodes does not eliminate the variable run times). |
Retried running at C1152 grid with
It ran to completion in one hour and one minute. The files were visualized using |
Tried running a C3072 grid with
It ran successfully in 6 minutes. |
Ran a regional C3445 case on Hera using the 'orog' files here: Set
It ran successfully in 16 minutes. |
The timing tests above were done to determine whether lower resolution versions of the new 'viirs' and 'bnu' data are needed. As long as a developer has access to a machine with large memory, only the high-resolution (30s) is needed. On Hera, users can request 'bigmem' nodes. And WCOSS2 has sufficient memory as well. However, users trying to run on their laptop could have problems. @barlage - I recommend we don't create lower-resolution versions of these files right now. They can always be created later. |
So the following snow contaminated files should be removed from all machines: Under
Under
The updated files (without snow contamination) should be added somewhere under
|
Tagging @WalterKolczynski-NOAA |
@GeorgeGayno-NOAA Please stage a new, complete sfc_climo directory on Hera and I will copy it to the authoritative fix locations as |
@GeorgeGayno-NOAA @HelinWei-NOAA Is this NOAA-EMC/global-workflow#1617 or separate? If it is separate, there should be a g-w issue for it. Sorry I didn't lead with that, there have been multiple simultaneous fix requests and I got them mixed up. |
Note: I am on leave next week, so if we can't get this done Friday, it will sit for a week. |
@barlage What is your opinion? Are all data ready? |
It is separate. NOAA-EMC/global-workflow#1617 is for soil color, which is not the same as soil type. |
@HelinWei-NOAA there was a minor issue encountered yesterday with the processing of the high res data in sfc_climo_gen, namely the number of processors required to generate grids with the high res data is more than can be handled with the very low res slope data (limitation of ESMF?). @GeorgeGayno-NOAA has a potential fix but @sanatcumar still needs to generate all the tiled files (sanath - you may need to create these in parallel to speed up the process). I'm hopeful that we can still get this in before @WalterKolczynski-NOAA goes on leave. |
Okay, then please create a new global-workflow issue using the "Fix File Update" template. |
Successfully tested this version of sfc_climo_gen on Hera. Regenerated a set of surface fields with the following in /util/sfc_climo_gen/sfc_gen.sh in /ush/sfc_climo_gen.sh results at |
@barlage and @sanatcumar - This is just about ready to merge. Should we merge this before or after #825? |
as required by the new 30s data. Fixes ufs-community#821.
viirs data. Fixes ufs-community#821.
The new data was tested on a C96 global uniform grid. The branch at 397cd43 was used on Hera. A local copy of the ./fix/sfc_climo` data was created under my personal directory. This directory contained the new veg, soil and substrate temperature data. To test the current data (with the spurious snow points), the
|
Results of the above test: All vegetation type files were identical except for tile6. Using the
The differences were confined to Antarctica. The new data has category '11' (Permanent wetlands). That does not make sense. @barlage FYI. |
Differences between the current and new substrate temperature data were larger. The new data is a remapping of the current global 2.6x1.5 degree dataset to a 0.5-degree dataset. The Here is a plot of tile 3 - using the new data, current data, and the new minus the current. In the broad sense, the plots are very similar. But near the poles, the differences can approach 2 degrees @sanatcumar and @barlage - I rechecked my remapping procedure and I don't see anything I am doing wrong. Should we be concerned about these differences? Or should we implement with the idea that we will replace this dataset (which is probably about 30 years old) with a newer and higher resolution dataset. Recall, the only reason for remapping to 0.5-degree is to avoid that quirky ESMF library issue. |
I think the problem may be a mistake in the geo referencing in the current file, not the new file. Investigating. |
After some investigation, the latitude records in the official substrate temperature file - The grib1 version of that file has this geo-referencing:
The latitude of the first point is 90.0. In the official netcdf version used by sfc_climo_gen, the first few latitudes are:
So, the data are shifted to the south by about 0.7 degrees. |
I recreated the OPS substrate temperature file using this code on Hera - The latitudes now match the grib header:
|
The new 0.5-degree version of the substrate data was created using a two-step method. First, the Then, the 0.5-degree grib1 file was converted to netcdf. The latitudes are correct:
|
Good catch. |
Run a quick test with the branch at 5ea89bc. Use a C96 global uniform grid and the corrected OPS substrate data. Then, repeat the test using the 0.5-degree data. The difference on tile3 is now much smaller and makes sense. The 0.5-degree data is correct and should be implemented. |
The 'v2' versions of the soil and veg type data still had problems with spurious glaciers. So a version 3 of these files was created. Updated (and hopefully final) list of files that should be added or removed from all machines: The following snow contaminated files should be removed: Under
Under
The updated soil and vegetation files (without snow contamination) should be added somewhere under
Also, a new higher resolution version of the 2.6x1.5 degree soil substrate temperature should also be added:
|
The 'v3' versions of the soil and vegetation type data were tested by @sanatcumar. All output looked correct. |
that contains the new soil and veg files. Fixes ufs-community#821.
A new version of the VIIRS vegetation type data was created that removes spurious snow points in warm regions.
On hera: /scratch2/NCEPDEV/land/data/input_data/vegetation_type/vegetation_type.viirs.v3.igbp.30s.nc
A new version of the BNU soil type data was created that is compatible with the updated vegetation data. It also includes additional records for the percentage and sand and clay.
On hera: /scratch2/NCEPDEV/land/data/input_data/soil_type/soil_type.bnu.v3.30s.nc
Test these new data in
sfc_climo_gen
and regenerate all GFS grids (including new soil color field).Using 30s data requires more MPI tasks. Because of a quirk in the ESMF library, using too many tasks with coarse input data, such as our substrate temperature data set (2.6x1.5 degree), will cause a library error. This can be resolved by using a higher resolution version (0.5-degree) of this data:
On hera: /scratch1/NCEPDEV/da/George.Gayno/ufs_utils.git/UFS_UTILS/fix/sfc_climo.test/substrate_temperature.gfs.0.5.nc
(Code to create the new 0.5-degree data on Hera:
/scratch1/NCEPDEV/da/George.Gayno/ufs_utils.git/climo_fields_netcdf/sorc/fv3.tbot.gfs.hires.netcdf
Related to #803.
The text was updated successfully, but these errors were encountered: