DAS-2214: Add the pre-commit file and update readme. (#86)

nasa · Aug 6, 2024 · d152d98 · d152d98
1 parent 1e06094
commit d152d98
Show file tree

Hide file tree

Showing 75 changed files with 2,631 additions and 1,879 deletions.
diff --git a/.git-blame-ignore-revs b/.git-blame-ignore-revs
@@ -0,0 +1,5 @@
+# For more information, see:
+# https://docs.github.com/en/repositories/working-with-files/using-files/viewing-a-file#ignore-commits-in-the-blame-view
+
+# Black code formatting of repository.
+50ff8f57feaff4471ce82274f34baafc3be9782f
diff --git a/.gitignore b/.gitignore
@@ -11,3 +11,4 @@ output/*
 .idea
 
 **/pulled-images.txt
+**/.python-version
diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
@@ -0,0 +1,22 @@
+ci:
+  autofix_prs: false
+repos:
+  - repo: https://github.com/pre-commit/pre-commit-hooks
+    rev: v4.6.0
+    hooks:
+      - id: trailing-whitespace
+      - id: end-of-file-fixer
+      - id: check-json
+      - id: check-yaml
+      - id: check-added-large-files
+  - repo: https://github.com/astral-sh/ruff-pre-commit
+    rev: v0.4.9
+    hooks:
+      - id: ruff
+        args: ["--fix", "--show-fixes"]
+  - repo: https://github.com/psf/black-pre-commit-mirror
+    rev: 24.4.2
+    hooks:
+      - id: black-jupyter
+        args: ["--skip-string-normalization"]
+        language_version: python3.11
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -0,0 +1,15 @@
+# CHANGELOG
+
+The harmony-regression-tests repository does not follow semantic
+versioning. Rather than a static releases, this repository contains of a number
+of regression tests that are each semi-independent.  This CHANGELOG file should be used
+to document pull requests to this repository.
+
+
+## 2024-08-05 ([#86](https://github.com/nasa/harmony-regression-tests/pull/86))
+
+Adds this file to capture changes to the repository.
+
+Adds pre-commit.ci behavior to the repository. This setup ensures consistent code style, catches common errors, and maintains file hygiene across the project.
+
+Updates the base image for all regression tests to `mambaorg/micromamba:1.5.8-jammy`
diff --git a/README.md b/README.md
@@ -8,7 +8,7 @@ UAT and Prod. This is the preferred method of verifying no regressons have
 occurred, when the services have been modified.
 
 Alternatively, each test can be run locally in a browser against SIT, UAT, PROD
-or localhost (harmony-in-a-box). This is a good choice for test development and
+or localhost (Harmony-In-A-Box). This is a good choice for test development and
 verifying service changes do not cause regression failures. Generally you run
 locally in the browser against a single service regression test.
 
@@ -53,15 +53,18 @@ created Zarr store.*
    to run after the run_notebooks command.  e.g. `./run_notebooks.sh hga n2z`
    would run the `harmony GDAL adapter` and `NetCDF-to-Zarr` regression tests.*
 
-1. *`HARMONY_HOST_URL` is the harmony base url for your target environment. e.g. `SIT` would be `https://harmony.sit.earthdata.nasa.gov`*
+1. *`HARMONY_HOST_URL` is the harmony base url for your target
+   environment. e.g. `SIT` would be `https://harmony.sit.earthdata.nasa.gov`*
 
-1. The `run_notebooks.sh` script cannot be used to test against Harmony-in-a-Box,
-   i.e. `HARMONY_HOST_URL=http://localhost:3000`, due to Docker-in-Docker issues.
-   To test against a local Harmony instance, the notebook should be run
-   manually on a Jupyter notebook server (e.g., in a browser).
+1. *The `run_notebooks.sh` script cannot be used to test against
+   Harmony-in-a-Box, i.e. `HARMONY_HOST_URL=http://localhost:3000`, due to
+   Docker-in-Docker issues.  To test against a local Harmony instance, the
+   notebook should be run manually on a Jupyter notebook server (e.g., in a
+   browser).*
+
+For more information on running a local Harmony instance, see the [Harmony
+README](https://github.com/nasa/harmony/blob/main/README.md).
 
-For more information on running a local Harmony instance, see:
-<https://github.com/nasa/harmony/blob/main/README.md>.
 
 ### Test in a Browser:
 
@@ -248,3 +251,36 @@ if the new image is named `ghcr.io/nasa/regression-tests-foo`, then we would add
 
 The `run_notebooks.sh` file can be used as described above to run the test suite. Notebooks are
 expected to exit with a non-zero exit code on failure when run from `papermill`.
+
+## pre-commit hooks:
+
+This repository uses [pre-commit](https://pre-commit.com/) to enable pre-commit
+checking the repository for some coding standard best practices. These include:
+
+* Removing trailing whitespaces.
+* Removing blank lines at the end of a file.
+* Ensure JSON files have valid formats.
+* [ruff](https://github.com/astral-sh/ruff) Python linting checks.
+* [black](https://black.readthedocs.io/en/stable/index.html) Python code
+  formatting checks.
+
+To enable these checks:
+
+```bash
+# Install pre-commit Python package:
+pip install pre-commit
+
+# Install the git hook scripts:
+pre-commit install
+```
+
+
+If you have installed the hooks locally, when you commit your changes the hook
+will validate your changes before actually committing to your repository. If
+there are failures you will have to opportunity to fix them and add them to
+your commit.
+
+[pre-commit.ci](pre-commit.ci) is configured such that these same hooks will be
+automatically run for every pull request. Because of this, it is highly
+recommended that you also do this locally, since failures will prevent your PR
+from being merged.
diff --git a/test/Dockerfile b/test/Dockerfile
@@ -1,6 +1,4 @@
-# Uses focal image because newer jammy image does not work with old docker versions
-# see related information: https://github.com/adoptium/containers/issues/215#issuecomment-1142046045
-FROM mambaorg/micromamba:1.4.2-focal
+FROM mambaorg/micromamba:1.5.8-jammy
 USER root
 
 ARG sub_dir

diff --git a/test/geoloco/Geoloco_Regression.ipynb b/test/geoloco/Geoloco_Regression.ipynb
@@ -33,9 +33,16 @@
     "\n",
     "from harmony import Client, Collection, Environment, Request\n",
     "\n",
-    "from utilities import (submit_and_download, get_dim_sizes, get_sds_data,\n",
-    "                       remove_results_files, print_error, print_success,\n",
-    "                       compare_dimensions, compare_data)"
+    "from utilities import (\n",
+    "    submit_and_download,\n",
+    "    get_dim_sizes,\n",
+    "    get_sds_data,\n",
+    "    remove_results_files,\n",
+    "    print_error,\n",
+    "    print_success,\n",
+    "    compare_dimensions,\n",
+    "    compare_data,\n",
+    ")"
    ]
   },
   {
@@ -89,10 +96,12 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "host_environment = {'http://localhost:3000': Environment.LOCAL,\n",
-    "                    'https://harmony.sit.earthdata.nasa.gov': Environment.SIT,\n",
-    "                    'https://harmony.uat.earthdata.nasa.gov': Environment.UAT,\n",
-    "                    'https://harmony.earthdata.nasa.gov': Environment.PROD}\n",
+    "host_environment = {\n",
+    "    'http://localhost:3000': Environment.LOCAL,\n",
+    "    'https://harmony.sit.earthdata.nasa.gov': Environment.SIT,\n",
+    "    'https://harmony.uat.earthdata.nasa.gov': Environment.UAT,\n",
+    "    'https://harmony.earthdata.nasa.gov': Environment.PROD,\n",
+    "}\n",
     "\n",
     "\n",
     "harmony_environment = host_environment.get(harmony_host_url)\n",
@@ -124,32 +133,42 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "mod021km_non_production_info = {'collection' : Collection(id='C1256826282-LAADSCDUAT'),\n",
-    "                               'granule_id' : 'G1259320275-LAADSCDUAT',\n",
-    "                               'variable' : ['EV_250_Aggr1km_RefSB'],\n",
-    "                               'downscale_size' : [0.01802, 0.01802]}\n",
-    "\n",
-    "mod35l2_non_production_info = {'collection' : Collection(id='C1257437479-LAADSCDUAT'),\n",
-    "                               'granule_id' : 'G1261599141-LAADSCDUAT',\n",
-    "                               'variable' : ['Cloud_Mask'],\n",
-    "                               'downscale_size' : [0.01802, 0.01802]}\n",
-    "\n",
-    "mod08d3_non_production_info = {'collection' : Collection(id='C1257773477-LAADSCDUAT'),\n",
-    "                               'granule_id' : 'G1259320277-LAADSCDUAT',\n",
-    "                               'variable' : ['Aerosol_Optical_Depth_Land_Ocean_Mean'],\n",
-    "                               'downscale_size' : [2, 2]}\n",
+    "mod021km_non_production_info = {\n",
+    "    'collection': Collection(id='C1256826282-LAADSCDUAT'),\n",
+    "    'granule_id': 'G1259320275-LAADSCDUAT',\n",
+    "    'variable': ['EV_250_Aggr1km_RefSB'],\n",
+    "    'downscale_size': [0.01802, 0.01802],\n",
+    "}\n",
+    "\n",
+    "mod35l2_non_production_info = {\n",
+    "    'collection': Collection(id='C1257437479-LAADSCDUAT'),\n",
+    "    'granule_id': 'G1261599141-LAADSCDUAT',\n",
+    "    'variable': ['Cloud_Mask'],\n",
+    "    'downscale_size': [0.01802, 0.01802],\n",
+    "}\n",
+    "\n",
+    "mod08d3_non_production_info = {\n",
+    "    'collection': Collection(id='C1257773477-LAADSCDUAT'),\n",
+    "    'granule_id': 'G1259320277-LAADSCDUAT',\n",
+    "    'variable': ['Aerosol_Optical_Depth_Land_Ocean_Mean'],\n",
+    "    'downscale_size': [2, 2],\n",
+    "}\n",
     "\n",
     "geo_proj4_string = '+a=6378137.0 +b=6356752.3142451793 +no_defs +proj=latlong'\n",
     "\n",
     "resampling_string = 'NN'\n",
     "\n",
-    "file_indicators = {'MOD021KM': 'EV_250_Aggr1km_RefSB_1.hdf',\n",
-    "                   'MOD35_L2': 'Cloud_Mask_1.hdf',\n",
-    "                   'MOD08_D3': 'Aerosol_Optical_Depth_Land_Ocean_Mean.hdf'}\n",
-    "\n",
-    "reference_data = {'MOD021KM': 'reference_data/MOD021KM.A2023001.0020.061.psrpcs_001701802061.EV_250_Aggr1km_RefSB_1.hdf',\n",
-    "                  'MOD35_L2': 'reference_data/MOD35_L2.A2023001.0020.061.psrpcs_001701881013.Cloud_Mask_1.hdf',\n",
-    "                  'MOD08_D3': 'reference_data/MOD08_D3.A2023001.061.psrpcs_001701881265.Aerosol_Optical_Depth_Land_Ocean_Mean.hdf'}"
+    "file_indicators = {\n",
+    "    'MOD021KM': 'EV_250_Aggr1km_RefSB_1.hdf',\n",
+    "    'MOD35_L2': 'Cloud_Mask_1.hdf',\n",
+    "    'MOD08_D3': 'Aerosol_Optical_Depth_Land_Ocean_Mean.hdf',\n",
+    "}\n",
+    "\n",
+    "reference_data = {\n",
+    "    'MOD021KM': 'reference_data/MOD021KM.A2023001.0020.061.psrpcs_001701802061.EV_250_Aggr1km_RefSB_1.hdf',\n",
+    "    'MOD35_L2': 'reference_data/MOD35_L2.A2023001.0020.061.psrpcs_001701881013.Cloud_Mask_1.hdf',\n",
+    "    'MOD08_D3': 'reference_data/MOD08_D3.A2023001.061.psrpcs_001701881265.Aerosol_Optical_Depth_Land_Ocean_Mean.hdf',\n",
+    "}"
    ]
   },
   {
@@ -167,28 +186,34 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "mod021km_geoloco_env = {Environment.LOCAL: mod021km_non_production_info,\n",
-    "                       Environment.UAT: mod021km_non_production_info,\n",
-    "                       Environment.SIT: mod021km_non_production_info}\n",
-    "mod35l2_geoloco_env = {Environment.LOCAL: mod35l2_non_production_info,\n",
-    "                       Environment.UAT: mod35l2_non_production_info,\n",
-    "                       Environment.SIT: mod35l2_non_production_info}\n",
-    "mod08d3_geoloco_env = {Environment.LOCAL: mod08d3_non_production_info,\n",
-    "                       Environment.UAT: mod08d3_non_production_info,\n",
-    "                       Environment.SIT: mod08d3_non_production_info}\n",
+    "mod021km_geoloco_env = {\n",
+    "    Environment.LOCAL: mod021km_non_production_info,\n",
+    "    Environment.UAT: mod021km_non_production_info,\n",
+    "    Environment.SIT: mod021km_non_production_info,\n",
+    "}\n",
+    "mod35l2_geoloco_env = {\n",
+    "    Environment.LOCAL: mod35l2_non_production_info,\n",
+    "    Environment.UAT: mod35l2_non_production_info,\n",
+    "    Environment.SIT: mod35l2_non_production_info,\n",
+    "}\n",
+    "mod08d3_geoloco_env = {\n",
+    "    Environment.LOCAL: mod08d3_non_production_info,\n",
+    "    Environment.UAT: mod08d3_non_production_info,\n",
+    "    Environment.SIT: mod08d3_non_production_info,\n",
+    "}\n",
     "\n",
     "if harmony_environment in mod021km_geoloco_env:\n",
-    "    mod021km_geoloco_info =  mod021km_geoloco_env[harmony_environment]\n",
+    "    mod021km_geoloco_info = mod021km_geoloco_env[harmony_environment]\n",
     "else:\n",
     "    mod021km_geoloco_info = None\n",
     "\n",
     "if harmony_environment in mod35l2_geoloco_env:\n",
-    "    mod35l2_geoloco_info =  mod35l2_geoloco_env[harmony_environment]\n",
+    "    mod35l2_geoloco_info = mod35l2_geoloco_env[harmony_environment]\n",
     "else:\n",
     "    mod35l2_geoloco_info = None\n",
     "\n",
     "if harmony_environment in mod08d3_geoloco_env:\n",
-    "    mod08d3_geoloco_info =  mod08d3_geoloco_env[harmony_environment]\n",
+    "    mod08d3_geoloco_info = mod08d3_geoloco_env[harmony_environment]\n",
     "else:\n",
     "    mod08d3_geoloco_info = None"
    ]
@@ -210,55 +235,69 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "if (mod021km_geoloco_info is not None and \n",
-    "    mod35l2_geoloco_info is not None and\n",
-    "    mod08d3_geoloco_info is not None):\n",
-    "\n",
-    "    mod021km_request = Request(collection=mod021km_geoloco_info['collection'],\n",
-    "                               granule_id=mod021km_geoloco_info['granule_id'],\n",
-    "                               variables=mod021km_geoloco_info['variable'],\n",
-    "                               scale_size=mod021km_geoloco_info['downscale_size'],\n",
-    "                               crs=geo_proj4_string,\n",
-    "                               interpolation=resampling_string)\n",
-    "\n",
-    "    mod35l2_request = Request(collection=mod35l2_geoloco_info['collection'],\n",
-    "                              granule_id=mod35l2_geoloco_info['granule_id'],\n",
-    "                              variables=mod35l2_geoloco_info['variable'],\n",
-    "                              scale_size=mod35l2_geoloco_info['downscale_size'],\n",
-    "                              crs=geo_proj4_string,\n",
-    "                              interpolation=resampling_string)\n",
-    "\n",
-    "    mod08d3_request = Request(collection=mod08d3_geoloco_info['collection'],\n",
-    "                              granule_id=mod08d3_geoloco_info['granule_id'],\n",
-    "                              variables=mod08d3_geoloco_info['variable'],\n",
-    "                              scale_size=mod08d3_geoloco_info['downscale_size'],\n",
-    "                              crs=geo_proj4_string,\n",
-    "                              interpolation=resampling_string)\n",
-    "\n",
-    "    mod021km_compare_file = submit_and_download(harmony_client, mod021km_request, file_indicators['MOD021KM'])\n",
-    "    mod35l2_compare_file = submit_and_download(harmony_client, mod35l2_request, file_indicators['MOD35_L2'])\n",
-    "    mod08d3_compare_file = submit_and_download(harmony_client, mod08d3_request, file_indicators['MOD08_D3'])\n",
+    "if (\n",
+    "    mod021km_geoloco_info is not None\n",
+    "    and mod35l2_geoloco_info is not None\n",
+    "    and mod08d3_geoloco_info is not None\n",
+    "):\n",
+    "\n",
+    "    mod021km_request = Request(\n",
+    "        collection=mod021km_geoloco_info['collection'],\n",
+    "        granule_id=mod021km_geoloco_info['granule_id'],\n",
+    "        variables=mod021km_geoloco_info['variable'],\n",
+    "        scale_size=mod021km_geoloco_info['downscale_size'],\n",
+    "        crs=geo_proj4_string,\n",
+    "        interpolation=resampling_string,\n",
+    "    )\n",
+    "\n",
+    "    mod35l2_request = Request(\n",
+    "        collection=mod35l2_geoloco_info['collection'],\n",
+    "        granule_id=mod35l2_geoloco_info['granule_id'],\n",
+    "        variables=mod35l2_geoloco_info['variable'],\n",
+    "        scale_size=mod35l2_geoloco_info['downscale_size'],\n",
+    "        crs=geo_proj4_string,\n",
+    "        interpolation=resampling_string,\n",
+    "    )\n",
+    "\n",
+    "    mod08d3_request = Request(\n",
+    "        collection=mod08d3_geoloco_info['collection'],\n",
+    "        granule_id=mod08d3_geoloco_info['granule_id'],\n",
+    "        variables=mod08d3_geoloco_info['variable'],\n",
+    "        scale_size=mod08d3_geoloco_info['downscale_size'],\n",
+    "        crs=geo_proj4_string,\n",
+    "        interpolation=resampling_string,\n",
+    "    )\n",
+    "\n",
+    "    mod021km_compare_file = submit_and_download(\n",
+    "        harmony_client, mod021km_request, file_indicators['MOD021KM']\n",
+    "    )\n",
+    "    mod35l2_compare_file = submit_and_download(\n",
+    "        harmony_client, mod35l2_request, file_indicators['MOD35_L2']\n",
+    "    )\n",
+    "    mod08d3_compare_file = submit_and_download(\n",
+    "        harmony_client, mod08d3_request, file_indicators['MOD08_D3']\n",
+    "    )\n",
     "\n",
     "    mod021km_test = True\n",
     "    mod35l2_test = True\n",
     "    mod08d3_test = True\n",
-    "    \n",
+    "\n",
     "    if compare_dimensions(reference_data['MOD021KM'], mod021km_compare_file):\n",
     "        if not compare_data(reference_data['MOD021KM'], mod021km_compare_file):\n",
     "            print_error('MOD021KM data mismatch.')\n",
     "            mod021km_test = False\n",
     "    else:\n",
     "        print_error('MOD021KM data dimension mismatch.')\n",
     "        mod021km_test = False\n",
-    "    \n",
+    "\n",
     "    if compare_dimensions(reference_data['MOD35_L2'], mod35l2_compare_file):\n",
     "        if not compare_data(reference_data['MOD35_L2'], mod35l2_compare_file):\n",
     "            print_error('MOD35_L2 data mismatch.')\n",
     "            mod35l2_test = False\n",
     "    else:\n",
     "        print_error('MOD35_L2 data dimension mismatch.')\n",
     "        mod35l2_test = False\n",
-    "    \n",
+    "\n",
     "    if compare_dimensions(reference_data['MOD08_D3'], mod08d3_compare_file):\n",
     "        if not compare_data(reference_data['MOD08_D3'], mod08d3_compare_file):\n",
     "            print_error('MOD08_D3 data mismatch.')\n",
@@ -275,9 +314,11 @@
     "        print_success('Geoloco Reprojection/Resampling/Regridding requests.')\n",
     "    else:\n",
     "        raise Exception('Geoloco test suite failed')\n",
-    "        \n",
+    "\n",
     "else:\n",
-    "    print(f'Geoloco is not configured for this environment: \"{harmony_environment}\" - skipping test.')"
+    "    print(\n",
+    "        f'Geoloco is not configured for this environment: \"{harmony_environment}\" - skipping test.'\n",
+    "    )"
    ]
   }
  ],