Metadata Specification Proposal #17
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,47 @@ | ||
| { | ||
| "RUN_CONFIG": { | ||
| "description": "Run Config for Hurricane Zach", | ||
| "Model": { | ||
| "realization": { | ||
| "Name": "CFE_SLOTH_realization", | ||
| "Type": "filesystem", | ||
| "Path": "/ngen/realization.json", | ||
| "Hash": "d2c6fbda93c134de495d69745fae11087784d2aa" | ||
| }, | ||
| "configuration": { | ||
| "Name": "CFE_SLOTH_ini", | ||
| "Type": "filesystem", | ||
| "Path": "/ngen/cfe_sloth.ini", | ||
| "Hash": "ab0c3dff59c4b282b172b90128159fda3386d012" | ||
| } | ||
| }, | ||
| "Forcings": { | ||
| "inputs": { | ||
| "Name": "Hurricane Zach", | ||
| "Type": "bucket", | ||
| "Path": "s3://awi-ciroh-ngen-data/AWI_001/forcings/", | ||
| "Hash": "220fff8bdd3b85f23d93e73b4bc7e3bc2c7c0f35" | ||
| }, | ||
| "Hydrofabric": { | ||
| "catchment": { | ||
| "Name": "Catchment(s) File v1.0", | ||
| "Type": "bucket", | ||
| "Path": "s3://awi-ciroh-ngen-data/AWI_001/catchments.geojson", | ||
| "Hash": "da39a3ee5e6b4b0d3255bfef95601890afd80709" | ||
| }, | ||
| "nexus": { | ||
| "Name": "Nexus File v1.0", | ||
| "Type": "bucket", | ||
| "Path": "s3://awi-ciroh-ngen-data/AWI_001/nexus.geojson", | ||
| "Hash": "cae054f62f697080d822fea9c7d9c268be8b7ac9" | ||
| }, | ||
| "crosswalk": { | ||
| "Name": "Crosswalk File v1.0", | ||
| "Type": "bucket", | ||
| "Path": "s3://awi-ciroh-ngen-data/AWI_001/crosswalk.geojson", | ||
| "Hash": "4c39964d1e30779f9992d3c00e94a39952cb102a" | ||
| } | ||
| } | ||
| } | ||
| } | ||
| } |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,182 +1,56 @@ | ||
| ### Reproducing | ||
|
|
||
| Get the files from the appropriate bucket & RUN_CONFIG | ||
| ``` | ||
| wget --no-parent https://awi-ciroh-ngen-data.s3.us-east-2.amazonaws.com/AWI_001/AWI_03W_113060_001.tar.gz . | ||
|
There was a problem hiding this comment. I would suggest using |
||
|
|
||
| tar -xvf AWI_03W_113060_001.tar.gz | ||
| ``` | ||
| Then we can confirm file location and integrity against the example JSON file. | ||
| ``` | ||
| { | ||
| "RUN_CONFIG": { | ||
| "description": "Run Config for AWI_03W_113060_001", | ||
| "Model": { | ||
| "realization": { | ||
|
There was a problem hiding this comment. Thanks for getting this going, @ZacharyWills! Having now stared at this for 30 minutes and gone back and forth about what comments I want to make, it seems clear to me that we could solve this problem in a lot of different ways. But, as will all solutions, they all have pros and cons. My biggest take away so far is that this initial proposal should set up future proposals for success. I think the way we do that is by going with the simplest data model that does not limit future additions in places where we want expansion. To get to that place, I think we first need some guard rails on what the spec is not. I figure we wont be able to determine all the things it is to start, without using it, so it is likely easier to say what it isn't. So kind of like goals, but anti-goals if you will haha. The purpose of saying what the spec isn't is to create consensus and avoid scope creep and have something to point to when suggestions are brought to the table. Please disagree and challenge me on this. It may just be an idea in my head and not a good one! My general thoughts so far are (numbered, so it is easier to respond. Not ordered):
I don't see it the same. I think you can make the case that you can have forcing over a given hydrofabric domain, so {
"Meta": {
"Hash": {
"Algo": "sha256"
}
}
}There was a problem hiding this comment. So the hierarchy is based off of the assumption that things within the workflow are somewhat dependent (generating forcings for only your selected hydrofabric is more efficient than always generating them everywhere) but there's a lot of discussion about that going on now here: owp-spatial/hfsubsetCLI#28 I totally agree on the syntax changes and I'll put some work in to fix the proposal to reflect those insights. For the most part I just wanted something to get the ball rolling on how Thanks again for taking a look at this and any input is so valuable at this point to get us going from 0 -> something. |
||
| "Name": "AWI_simplified_realization", | ||
| "Type": "filesystem", | ||
| "Path": "AWI_03W_113060_001/config/awi_simplified_realization.json", | ||
| "Hash": "792554dcf48b61120cfc648cc6711d2b5e61d321" | ||
| }, | ||
| "configuration": { | ||
| "Name": "CFE_SLOTH_ini", | ||
| "Type": "filesystem", | ||
| "Path": "AWI_03W_113060_001/config/awi_config.ini", | ||
| "Hash": "e8283864026040ce1ce5a7dca79b9f4f04744b47" | ||
| } | ||
| }, | ||
| "Forcings": { | ||
| "inputs": { | ||
| "Name": "Hurricane Zach", | ||
| "Type": "filesystem", | ||
| "Path": "AWI_03W_113060_001/forcings", | ||
| "Hash": "da39a3ee5e6b4b0d3255bfef95601890afd80709" | ||
| }, | ||
| "Hydrofabric": { | ||
| "catchment": { | ||
| "Name": "Catchment(s) File v1.0", | ||
| "Type": "bucket", | ||
| "Path": "AWI_03W_113060_001/config/catchment_data.geojson", | ||
| "Hash": "880feb145f254976600bd8968ef730105de6cbee" | ||
| }, | ||
| "nexus": { | ||
| "Name": "Nexus File v1.0", | ||
| "Type": "bucket", | ||
| "Path": "AWI_03W_113060_001/config/nexus_data.geojson", | ||
| "Hash": "86a029a15e7cf67bc69f2390038a74b69b09af04" | ||
| }, | ||
| } | ||
| } | ||
| } | ||
| } | ||
| ``` | ||
| Note that to recreate the sums for the focings file, I simply summed the files then piped the output to its own sum. | ||
| ``` | ||
| shasum -a 256 AWI_03W_113060_001/forcings/ | shasum | ||
| ``` | ||
|
There was a problem hiding this comment. General comments:
There was a problem hiding this comment. As for general formatting, what are your thoughts on switching to a table format similar to schema.org? A schema.org Model
There was a problem hiding this comment. I figure |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,26 @@ | ||
| --- CONFIG START --- | ||
|
There was a problem hiding this comment. I think we can remove this. It kind of implies that the listed fields need to be ordered (although, I think that is a stretch). |
||
| ### Model | ||
| | Name of Model files | type | PATH to .json file | HASH of file | | ||
| | ----------- | ----------- | ----------- | ----------- | | ||
| | CFE_SLOTH_realization | filesystem | /ngen/realization.json | d2c6fbda93c134de495d69745fae11087784d2aa | | ||
| | CFE_SLOTH_ini | filesystem | /ngen/cfe_sloth.ini | ab0c3dff59c4b282b172b90128159fda3386d012 | | ||
| ### Forcings | ||
| | Name of Forcings | type | PATH to forcings file(s) | HASH of file(s) | | ||
| | ----------- | ----------- | ----------- | ----------- | | ||
| | Hurricane Zach | bucket | s3://awi-ciroh-ngen-data/AWI_001/forcings/ | 220fff8bdd3b85f23d93e73b4bc7e3bc2c7c0f35 | | ||
| ### Hydrofabric | ||
| | Name of Hydrofabric | type | PATH to .json file | HASH of file | | ||
| | ----------- | ----------- | ----------- | ----------- | | ||
| | Catchment(s) File | bucket | s3://awi-ciroh-ngen-data/AWI_001/catchments.geojson | da39a3ee5e6b4b0d3255bfef95601890afd80709 | | ||
| | Nexus File | bucket | s3://awi-ciroh-ngen-data/AWI_001/nexus.geojson | cae054f62f697080d822fea9c7d9c268be8b7ac9 | | ||
| | Crosswalk File | bucket | s3://awi-ciroh-ngen-data/AWI_001/crosswalk.geojson | 4c39964d1e30779f9992d3c00e94a39952cb102a | | ||
|
|
||
| All values within each section are defined for the run to evaluate when a run is unique and the change that makes it so. For example changing the hash of any of the fields for the referenced files should change the RUN CONFIG into something new. | ||
|
|
||
| Note: the Realization, Catchment, Nexus and other model-required files must be searching by name. | ||
|
|
||
| | Valid | Not Valid | | ||
| | ----- | --------- | | ||
| | hurricane_marty_realization.json | hurricane_marty.json | | ||
| | Houston_catchments.geojson | Houston.geojson | | ||
| | Nexus_2012_flood.geojson | 2012_flood.geojson | | ||
|
There was a problem hiding this comment. Not sure if this file was intended to make it into the proposal? It seems like some of this content could be reshaped a bit and make its way into the README instead? Maybe we do want a living notes document though. Although, a github discussion thread might be more appropriate and accomplish the same goal. Thoughts? |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,23 @@ | ||
| # Notes on a "NextGen Universal Package" | ||
|
|
||
| There's been some discussion in the community about the combinatorial effects of opening up the possibility to build ever-more-complicated systems while making it more difficult to resproduce results in the community, or to share results for integration into our collective understanding of the world. | ||
|
|
||
|
|
||
| The first step to building that collective understanding is based on having the same interpretation and categorization of the data we have and will produce. | ||
|
|
||
| Firstly there must be a heirarchical underestanding of how we represent data and configurations. Note that these aren't 1:1 with the data and configurations, these are for the heirarchical framing of how those are organized. | ||
|
There was a problem hiding this comment. small typo! underestanding -> understanding |
||
|
|
||
| The identified components are as follows: | ||
|
|
||
| - _Data_ | ||
| - Hydrofabric | ||
| - Underlying physical understanding of the domain | ||
| - Forcings | ||
| - Input data over a temporal and spacial domain | ||
| - _Configuration_ | ||
| - Realization of model allocation | ||
| - Models (version of BMI-compatible model) | ||
| - Model configurations | ||
|
|
||
| Each Instance of the combination of these things is hereafter called a RUN and the grouping of all the aforementioned items necessitates a standardized RUN CONFIG. | ||
|
|
||
| A preliminary Specification for NextGen RUN CONFIGs has been proposed. | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I likely just dont know the context, so this might be a moot point, but what is a
RUN_CONFIG? Can we just remove it? If not, can we provide some supporting text to say what aRUN_CONFIGis?