DAS-1900: As a developer, I would like to be able to make an HTTPS request that is routed to my variable generation service, so that I can request UMM-Var records are generated in the cloud

[william-valencia]

Overview

What is the feature?

DAS-1900: As a developer, I would like to be able to make an HTTPS request that is routed to my variable generation service, so that I can request UMM-Var records are generated in the cloud

We decided that variable generation(earthdata-varinfo) will be routed through graphQL

What is the Solution?

I created an earthdata-varinfo lambda. Also modified the cmr-graphQL to accept a generateVariableDrafts query under the existing collection query, which will return variable drafts generated by the earthdata-varinfo lambda.

What areas of the application does this impact?

Collection query will have a new field(generateVariableDrafts) that can be requested.
Also added a new field for variableDrafts called metadataSpecification.

Testing

Reproduction steps

• Environment for testing:
• cmr-graphql
• Collection to test with:
• C1598621093-GES_DISC or similar

1. Navigate to Apollo Studio for the environment you will be working in
2. Request the following collection query:

query Collection($params: CollectionInput) {
  collection(params: $params) {
    conceptId
    generateVariableDrafts {
      count
      items {
        dataType
        definition
        dimensions
        longName
        name
        standardName
        units
        metadataSpecification
      }
    }
  }
}

4. Specify the variable input params conceptId ie:

{
  "params": {
    "conceptId": "C1598621093-GES_DISC"
  }
}

5. Put in your authorization token in the Headers area.
6. Submit the query and you should receive something like the following back:

{
  "data": {
    "collection": {
      "conceptId": "C1598621093-GES_DISC",
      "generateVariableDrafts": {
        "count": 16,
        "items": [
          {
            "dataType": "int32",
            "definition": "Grid/time",
            "dimensions": [
              {
                "Name": "Grid/time",
                "Size": 1,
                "Type": "TIME_DIMENSION"
              }
            ],
            "longName": "Grid/time",
            "name": "Grid/time",
            "standardName": "time",
            "units": "seconds since 1970-01-01 00:00:00 UTC",
            "metadataSpecification": {
              "URL": "https://cdn.earthdata.nasa.gov/umm/variable/v1.8.2",
              "Name": "UMM-Var",
              "Version": "1.8.2"
            }
          },...
}

Attachments

Please include relevant screenshots or files that would be helpful in reviewing and verifying this change.

Checklist

• I have added automated tests that prove my fix is effective or that my feature works
• New and existing unit tests pass locally with my changes
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• My changes generate no new warnings

[codecov]

Codecov Report

Merging #63 (d956e67) into main (9de0272) will not change coverage.
The diff coverage is 100.00%.

@@            Coverage Diff            @@
##              main       #63   +/-   ##
=========================================
  Coverage   100.00%   100.00%           
=========================================
  Files           63        65    +2     
  Lines         1304      1336   +32     
  Branches       176       180    +4     
=========================================
+ Hits          1304      1336   +32     

Files	Coverage Δ
src/constants/index.js	100.00% <ø> (ø)
src/datasources/collectionVariableDrafts.js	100.00% <100.00%> (ø)
src/datasources/orderOption.js	100.00% <ø> (ø)
src/resolvers/collection.js	100.00% <100.00%> (ø)
src/resolvers/variableDraft.js	100.00% <100.00%> (ø)
src/utils/aws/getLambdaConfig.js	100.00% <100.00%> (ø)
src/utils/parseRequestedFields.js	100.00% <100.00%> (ø)

[macrouch]

Is there anything special you need to do to get the python lambda working? Does serverless-python-requirements do everything? Is there a specific python install required on your dev system? I think the readme needs updating with information about the new capability and any specific python information

[william-valencia]

Is there anything special you need to do to get the python lambda working? Does serverless-python-requirements do everything? Is there a specific python install required on your dev system? I think the readme needs updating with information about the new capability and any specific python information

Yes I did run the commands below to get things working. Will update the README
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

[owenlittlejohns]

Nice work. Most of my comments are small, including:

• Trying out a Python 3.11 runtime.
• Cleaning up the requirements.txt to only include 3rd party packages explicitly imported in handler.py.
• The context manager thing for the temporary directory clean-up.
• Running a Python linter/style checker over handler.py to catch some minor things.

Potentially a more important thing is that there is no unit testing of handler.py. I think that's because this is the fist instance of Python in this repository. Not sure what @abbottry thinks about that (given the high test coverage he was just touting).

The only other thing is a PSA about the token handling. I'll try and get that consistent in earthdata-varinfo (for search/download vs the publication) for the next version, which will mean you have a simpler life, albeit with a couple of code changes in the next PR for this repo.

[abbottry]

@owenlittlejohns we absolutely need python tests, you're right! We should also tie that into GitHub actions.

[william-valencia]

Added a python unit test.
From src/earthdataVarinfo
Need to run
python3 -m unittest discover
...

Ran 3 tests in 0.023s

OK

[owenlittlejohns]

Thanks for adding the unit tests. I think the only thing missing from them is plugging them in to the CI/CD workflow so that they run for PRs, etc. (Unless I've missed something here)

[eni-awowale]

As a person who is new to GraphQL is it possible we could have something that links to a "NASA How To for GraphQL". This ReadMe feels a bit dense. I can't tell what I need to install and how to get there and what part is actually related to earthdata-varinfo. It seems like the section for "Generate Collection Variable Drafts", is what is applicable here but it doesn't come off really clearly to me.

[eni-awowale]

I think most of this looks good. I didn't really give too much feedback on the GraphQL portions because I am not too familiar with it.

[owenlittlejohns]

Thanks for plumbing in the tests to the CI/CD. Nice work!

[macrouch]

Only requesting changes to block merging, so I can do a test deployment