Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Migrate to new metadata format step 1 #205

Merged
merged 21 commits into from
Aug 1, 2019
Merged

Migrate to new metadata format step 1 #205

merged 21 commits into from
Aug 1, 2019

Conversation

climbfuji
Copy link
Collaborator

@climbfuji climbfuji commented Jul 25, 2019 •
edited
Loading

This PR and associated PRs enable parsing metadata in the new metadata format on a file-by-file basis. All type and variable definitions in FV3 are converted to the new metadata with the correct dimensions filled in. In addition, one physics scheme (ysuvdif.F90) is also converted to the new metadata.

Changes for ccpp-framewqork:

  • remove old logic to identify kind and type definitions from mkcap.py
  • bugfixes to new metadata parser to correctly interpret optional attributes and to allow uppercase characters in variable standard names
  • improve metadata converter:
    • suppress optional argument for variable/kind/type definitions, do not convert standard names to lowercase strings
    • add host-model dependent information and add an additional conversion script for schemes that leverages information on array dimensions from type/variable definition metadata in new format
    • add a simplified version of the metadata converter for schemes that uses the type/variable definition metadata (in new format) to fill in the missing information on array dimensions
  • add functionality to detect and parse new metadata with ccpp_prebuild.py on a per-file basis

climbfuji added 6 commits July 25, 2019 16:03
@climbfuji
scripts/convert_metadata.py: don't convert standard names to lower ca…
d7824f9 
…se, skip optional and intent attributes for variable/type definitions
@climbfuji
scripts/metadata_parser.py: add functionality to parse new metadata f…
9bf55e6 
…ormat and convert back into old format, currently tested for schemes and module variable definitions, not yet for DDTs
@climbfuji
scripts/metavar.py: bugfix in routine valid_value to return the corre…
3b531f0 
…ct answer when checking logicals
@climbfuji
scripts/mkcap.py: bugfix, perform type_kind_test correctly for both t…
c8423d2 
…ype and kind definitions when initializing from a metadata table
@climbfuji
scripts/parse_tools/parse_checkers.py: allow uppercase characters in …
3216854 
…standard names
@climbfuji
scripts/metadata_table.py: make script executable
3157123
@climbfuji
Copy link
Collaborator Author

Associated PRs:

https://github.com/NCAR/FV3/pull/181
#205
NCAR/ccpp-physics#288
NEMSfv3gfs: not yet created (only update of submodule pointers?)

@climbfuji climbfuji mentioned this pull request Jul 25, 2019
Merged
gold2718
gold2718 approved these changes Jul 28, 2019
View reviewed changes
Copy link
Collaborator

@gold2718 gold2718 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks okay, could use some improvement.

scripts/metavar.py Outdated Show resolved Hide resolved
scripts/parse_tools/parse_checkers.py Outdated Show resolved Hide resolved
climbfuji added 3 commits July 29, 2019 12:54
@climbfuji
scripts/metadata_table.py: adjust indents, allow literals in local va…
dc1cbca 
…riable names that denote the start of a new variable block
@climbfuji
scripts/metavar.py: use __true_vals and __false_vals
0dd8c35
@climbfuji
scripts/parse_tools/__init__.py, scripts/parse_tools/parse_checkers.p…
8c8ac84 
…y: define and export LITERAL, use in logical expressions for array references
@climbfuji climbfuji mentioned this pull request Jul 31, 2019
Closed
climbfuji and others added 7 commits July 31, 2019 13:44
@climbfuji
scripts/ccpp_prebuild.py, scripts/common.py, scripts/metadata_parser.…
9bd830f 
…py: add capability to parse variable and type definition metadata tables in new metadata format on a per-file basis
@climbfuji
scripts/mkcap.py: remove unnecessary type_kind_var logic and legacy g…
9eac6a1 
…uard to prevent using multi-dimensional character arrays
@climbfuji
scripts/mkstatic.py: remove unnecessary import statement
a640f0a
@climbfuji
scripts/metavar.py: do not check for derived data types being registe…
2bda79b 
…red, this only works if all DDTs are defined in the same file and in the correct order
@climbfuji
scripts/convert_metadata.py: add logic and host-model dependent adjus…
3ad6131 
…tments to convert type/variable definitions and fill in dimensions, array references, ...
Allow comments in type definition lines
4d72408
@climbfuji
scripts/convert_metadata_schemes_using_typedef_dims.py: convert schem…
4b61112 
…e tables one by one, filling in dimension information from type definitions
@climbfuji climbfuji mentioned this pull request Jul 31, 2019
Merged
@climbfuji
Copy link
Collaborator Author

See https://github.com/NCAR/NEMSfv3gfs/pull/210 for regression testing.

@climbfuji climbfuji marked this pull request as ready for review August 1, 2019 13:46
llpcarson
llpcarson approved these changes Aug 1, 2019
View reviewed changes
Copy link
Contributor

@llpcarson llpcarson left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved

@climbfuji climbfuji merged commit 8734529 into NCAR:gmtb/develop Aug 1, 2019
This was referenced Aug 18, 2021
Merged
Merged
@climbfuji climbfuji deleted the new_metadata_format_step1_20190725 branch June 27, 2022 03:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@llpcarson llpcarson llpcarson approved these changes

@gold2718 gold2718 gold2718 approved these changes

@grantfirl grantfirl Awaiting requested review from grantfirl

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@climbfuji @gold2718 @llpcarson