docs/guides/configuring-harmony-service.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Configuring a Harmony Service\n",
    "\n",
    "**Note:** If you are developing a new serivce, it is highly recommended that you start by reading the instructions at `docs/guides/adapting-new-services.md` and then running the `bin/generate-new-service` script in the [NASA Harmony repository](https://github.com/nasa/harmony). This will take care of much of the boilerplate configuration/code covered in this notebook.\n",
    "\n",
    "This notebook will show the steps required to configure a [Harmony](https://harmony.earthdata.nasa.gov/) service, covering the following points:\n",
    "\n",
    "* Initial configuration of a new service in Harmony.\n",
    "* Associating a collection with an existing service.\n",
    "* Enabling service discovery via Earthdata Search Client ([EDSC](https://search.earthdata.nasa.gov/search)).\n",
    "* Associating variables with an existing collection.\n",
    "\n",
    "\n",
    "The following requirements are assumed to be already fulfilled:\n",
    "\n",
    "* A Docker image, containing a service that is wrapped in a `HarmonyAdapter` instance, exists in a place that can be accessed by Harmony.\n",
    "* A collection, containing granules, has been ingested and has associated UMM-C and UMM-G records.\n",
    "* Write access to the CMR provider containing the collection to be associated with the new Harmony service.\n",
    "* Access to the [NASA harmony repository](https://github.com/nasa/harmony), including the ability to push branches to the remote repository, and to open pull requests (PRs).\n",
    "\n",
    "The cell below will import the packages required for this notebook - it will need to be run ahead of most of the cells below that make requests against the CMR and the UMM-Var Generator (UVG) APIs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Install prerequisite packages\n",
    "import sys\n",
    "\n",
    "!{sys.executable} -m pip install requests"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "import xml.etree.ElementTree as ET\n",
    "import requests"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Collection and granule record requirements:\n",
    "\n",
    "Harmony uses records in the Common Metadata Repository ([CMR](https://cmr.earthdata.nasa.gov/search)) to determine which services are to be used for which collection. There are several Unified Metadata Model (UMM) record types that are used:\n",
    "\n",
    "* [UMM-C](https://git.earthdata.nasa.gov/projects/EMFD/repos/unified-metadata-model/browse/collection): Collection records, which contain information describing the collection itself, such as native file information, instrumentation or observing campaign used to take the data.\n",
    "* [UMM-G](https://git.earthdata.nasa.gov/projects/EMFD/repos/unified-metadata-model/browse/granule): Granule records, with specific information on an individual file within a collection. For example, spatial or temporal extents.\n",
    "* [UMM-Var](https://git.earthdata.nasa.gov/projects/EMFD/repos/unified-metadata-model/browse/variable): Variable records, detailing individual variables that are common to granules within the same collection. For example, sea surface temperature, or longitude.\n",
    "* [UMM-S](https://git.earthdata.nasa.gov/projects/EMFD/repos/unified-metadata-model/browse/service): Service records, that provide information on a back-end service that can be called to retrieve or transform hosted data. These records also help EDSC to configure the options presented to their users in order to make a valid request to these services.\n",
    "\n",
    "Before proceeding further, you should have granules ingested by a [Cumulus](https://github.com/nasa/cumulus) instance. These granules should each have a UMM-G record and be within a collection with a UMM-C record, hosted in a CMR provider. It is not required for Harmony to utilize cloud-hosted data ingested via Cumulus, but it is strongly recommended to reduce egress cost.\n",
    "\n",
    "First make a note of the collection concept ID for the cloud-hosted collection you want associated with a Harmony service. It has the format \"C1234567890-PROVIDER\", where \"PROVIDER\" corresponds to your CMR provider."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "base_cmr_url = 'https://cmr.uat.earthdata.nasa.gov'  # Update this value to the correct environment\n",
    "base_uvg_url = 'https://uvg.uat.earthdata.nasa.gov'  # Update this value to the correct environment (to use UVG)\n",
    "collection_concept_id = 'C1234567890-PROVIDER'  # Update this value to that of your collection\n",
    "provider = 'PROVIDER'  # Update this value to your provider"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, ensure that the UMM-G records in your collection contain the required entry in the `RelatedUrls` field. This might look as follows:\n",
    "\n",
    "```json\n",
    "{\n",
    "    ...,\n",
    "    \"RelatedUrls\": [\n",
    "        {\n",
    "            \"URL\": \"https://www.cloud-provider.com/path/to/granule/file.nc4\",\n",
    "            \"Type\": \"GET DATA\"\n",
    "        },\n",
    "        {\n",
    "            \"URL\": \"https://opendap.earthdata.nasa.gov/collections/C1234567890-PROVIDER/granules/granuleUR\",\n",
    "            \"Type\": \"USE SERVICE API\",\n",
    "            \"Subtype\": \"OPENDAP DATA\"\n",
    "        }\n",
    "    ],\n",
    "    ...\n",
    "}\n",
    "```\n",
    "\n",
    "Alternatively, the Atom JSON format of the same granule record would look like:\n",
    "\n",
    "```json\n",
    "{\n",
    "    ...,\n",
    "    \"links\": [\n",
    "        {\n",
    "            \"rel\": \"http://esipfed.org/ns/fedsearch/1.1/data#\",\n",
    "            \"title\": \"Files may be downloaded directly to your workstation from this link\",\n",
    "            \"hreflang\": \"en-US\",\n",
    "            \"href\": \"https://www.cloud-provider.com/path/to/granule/file.nc4\"\n",
    "        },\n",
    "        {\n",
    "            \"rel\": \"http://esipfed.org/ns/fedsearch/1.1/service#\",\n",
    "            \"title\": \"OPeNDAP request URL (GET DATA : OPENDAP DATA)\",\n",
    "            \"hreflang\": \"en-US\",\n",
    "            \"href\": \"https://opendap.uat.earthdata.nasa.gov/collections/C1234567890-PROVIDER/granules/granuleUR\"\n",
    "        }\n",
    "    ],\n",
    "    ...\n",
    "}\n",
    "```\n",
    "\n",
    "Harmony retrieves the Atom JSON response for granule record, and currently will retrieve the URL of the first link with the correct `rel` type. A user can also specify a string literal pattern that must be present in that URL, for example \"opendap\", to ensure a specific URL is retrieved."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you intend for Harmony job results that include this collection to be shareable, make sure that guests have `read` permission on the collection (via [CMR ACLs endpoints](https://cmr.earthdata.nasa.gov/access-control/site/docs/access-control/api.html)), and if no EULAs are present that the `harmony.has-eula` tag is associated with the collection and set to `false` via the CMR `/search/tags/harmony.has-eula/associations` endpoint. Example request body: `[{\"concept_id\": \"C1233860183-EEDTEST\", \"data\": false}]`. All collections used in the Harmony job must meet these two requirements in order for the job to be shareable."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Activating a service:\n",
    "\n",
    "At this point you should have a collection of granules with UMM-C and UMM-G records. Additionally, a Docker image of your service should be hosted in a repository that is accessible to Harmony. This could be the Harmony AWS instance Elastic Container Repository (ECR), or a public DockerHub account, for example.\n",
    "\n",
    "The following step will describe a pull request (PR) that should be made against the [NASA Harmony repository](https://github.com/nasa/harmony) in order to activate your service. Once this PR has been merged into the repository, and deployed to the relevant environment, it should be possible to make Harmony requests using your service against the configured collections by either constructing a Harmony URL manually or using the [harmony-py](https://pypi.org/project/harmony-py/) Python package.\n",
    "\n",
    "To activate a new service, you will need to include two things in the PR:\n",
    "\n",
    "* Environment variables for the service in [env-defaults](https://github.com/nasa/harmony/blob/main/env-defaults).\n",
    "* An entry in the [services.yml](https://github.com/nasa/harmony/blob/main/config/services.yml) configuration file.\n",
    "\n",
    "#### Environment variables for the service:\n",
    "\n",
    "HARMONY_SERVICE_EXAMPLE_IMAGE=harmonyservices/service-example:latest\n",
    "HARMONY_SERVICE_EXAMPLE_REQUESTS_CPU=128m\n",
    "HARMONY_SERVICE_EXAMPLE_REQUESTS_MEMORY=128Mi\n",
    "HARMONY_SERVICE_EXAMPLE_LIMITS_CPU=128m\n",
    "HARMONY_SERVICE_EXAMPLE_LIMITS_MEMORY=512Mi\n",
    "HARMONY_SERVICE_EXAMPLE_INVOCATION_ARGS='python -m harmony_service_example'\n",
    "\n",
    "The REQUESTS_CPU, REQUESTS_MEMORY, LIMITS_CPU, and LIMITS_MEMORY parameters are used for configuring the needed resources for running the docker container in a pod on kubernetes. See https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/ for details.\n",
    "\n",
    "**A chained Harmony request:**\n",
    "\n",
    "As mentioned above, a request to Harmony could involve multiple services, for example, the first service could extract a subset of variables from a granule hosted in OPeNDAP, while the second service could mask the retrieved variables using a GeoJSON polygon. There are examples of chained workflows in [config/services.yml](https://github.com/nasa/harmony/blob/main/config/services.yml) (any service with multiple `steps:`).\n",
    "\n",
    "\n",
    "\n",
    "#### A `services.yml` entry:\n",
    "\n",
    "There is additional documentation for adding a new entry to the [config/services.yml](https://github.com/nasa/harmony/blob/main/config/services.yml) file available [here](https://github.com/nasa/harmony/blob/main/docs/guides/adapting-new-services.md#5-registering-services-in-servicesyml).\n",
    "\n",
    "You will be required to add a unique entry to the `services.yml` for each service in each environment. Each service (or service chain) must be represented by one and only one unique umm-s concept. Any collections that support this service (or service chain) need to be associated with the umm-s concept. Here is an example service template from the documentation:\n",
    "\n",
    "```yaml\n",
    "- name: harmony/service-example    # A unique identifier string for the service, conventionally <team>/<service>\n",
    "  data_operation_version: '0.17.0' # The version of the data-operation messaging schema to use\n",
    "  has_granule_limit: true          # Optional flag indicating whether we will impose granule limts for the request. Default to true.\n",
    "  default_sync: false              # Optional flag indicating whether we will force the request to run synchrously. Default to false.\n",
    "  type:                            # Configuration for service invocation\n",
    "      <<: *default-turbo-config    # To reduce boilerplate, services.yml includes default configuration suitable for all Docker based services.\n",
    "      params:\n",
    "        <<: *default-turbo-params  # Always include the default parameters for docker services\n",
    "        env:\n",
    "          <<: *default-turbo-env   # Always include the default docker environment variables and then add service specific env\n",
    "          STAGING_PATH: public/harmony/service-example # The S3 prefix where artifacts generated by the service will be stored\n",
    "  umm_s: S1234-EXAMPLE            # Service concept id for the service. It is a required field and must be a string.\n",
    "  collections:                    # Optional, should not exist in most cases. It is only used when there are granule_limit or variables applied to collections of the service.\n",
    "    - id: C1234-EXAMPLE\n",
    "      granule_limit: 1000         # A limit on the number of granules that can be processed for the collection (OPTIONAL - defaults to no limit)\n",
    "      variables:                  # A list of variables provided by the collection (OPTIONAL)\n",
    "        - v1\n",
    "        - v2\n",
    "  maximum_sync_granules: 1        # Optional limit for the maximum number of granules for a request to be handled synchronously. Defaults to 1. Set to 0 to only allow async requests.\n",
    "  capabilities:                   # Service capabilities\n",
    "    subsetting:\n",
    "      bbox: true                  # Can subset by spatial bounding box\n",
    "      temporal: true              # Can subset by a time range\n",
    "      variable: true              # Can subset by UMM-Var variable\n",
    "      multiple_variable: true     # Can subset multiple variables at once\n",
    "    output_formats:               # A list of output mime types the service can produce\n",
    "      - image/tiff\n",
    "      - image/png\n",
    "      - image/gif\n",
    "    reprojection: true            # The service supports reprojection\n",
    "  steps:\n",
    "      - image: !Env ${QUERY_CMR_IMAGE} # The image to use for the first step in the chain\n",
    "      - image: !Env ${HARMONY_EXAMPLE_IMAGE}     # The image to use for the second step in the chain\n",
    "      \n",
    "- name: harmony/http-example  # An example of configuring the HTTP backend\n",
    "  type:\n",
    "    name: http                # This is an HTTP endpoint\n",
    "    params:\n",
    "      url: http://www.example.com/harmony  # URL for the backend service\n",
    "  # ... And other config (collections / capabilities) as in the above docker example\n",
    "```\n",
    "\n",
    "In a chained workflow there are multiple `steps` added to the template. These are in the order they should be invoked.\n",
    "\n",
    "#### Making a PR:\n",
    "\n",
    "Once you have a git branch with a workflow template and the necessary entries in the `config/services.yml` file, you should open a pull request to merge those changes into the NASA Harmony repository. Once merged, the changes will need to be deployed to the specified environments to activate your service. At that point you can begin making HTTP requests to retrieve output from your service via `harmony-py`, a browser, the Python `requests` package, cURL, or other client."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### UMM-S record:\n",
    "As mentioned above, an unique umm-s record needs to be created for each service (or service chain) and it is specified as the value of the umm_s key in services.yml configuration of the harmony service. \n",
    "\n",
    "When creating a UMM-S record, it is important to ensure you select a Harmony service, and that the URL the service points to is the base Harmony URL for the environment your service record relates to. See [UMM-S Guidance for Harmony Services](https://wiki.earthdata.nasa.gov/display/HARMONY/UMM-S+Guidance+for+Harmony+Services) for additional details on UMM-S curation for Earthdata Search discovery. Note, you will need to replicate this services record across environments in which the service will operate.\n",
    "\n",
    "#### Creating a UMM-S record in MMT\n",
    "\n",
    "Once signed in to MMT, you can click on the \"Manage Services\" button near the centre, followed by \"Create New Record\"\n",
    "\n",
    "![Create a new UMM-S record](../images/mmt_new_service_page.png \"Create a new UMM-S record\")\n",
    "\n",
    "As with the MMT interface for new UMM-Var records, this will take you to a multi-page form where you can specify the capabilities and requirements of your service. These options will inform EDSC the information required from users as input to the service. For example, if the service performs spatial subsetting, the user may need to provide the geographic values of a bounding box.\n",
    "\n",
    "#### Creating a UMM-S record via the CMR API:\n",
    "\n",
    "There is documentation on this process available [here](https://cmr.earthdata.nasa.gov/ingest/site/docs/ingest/api.html#create-update-service). Alternatively, you could create a UMM-S record via an HTTP PUT request. Note, the service metadata is an example - consult the full [schema](https://git.earthdata.nasa.gov/projects/EMFD/repos/unified-metadata-model/browse/service) for more options:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "headers = {'Content-type': 'application/vnd.nasa.cmr.umm+json',\n",
    "           'Echo-Token': echo_token,\n",
    "           'Accept': 'application/json'}\n",
    "base_cmr_url = 'https://cmr.uat.earthdata.nasa.gov'\n",
    "service_native_id = 'sample_native_id_value'  # This must be unique to the provider.\n",
    "\n",
    "service_metadata = {'Name': 'harmony-service-name',\n",
    "                    'Version': '0.9.0',\n",
    "                    'Description': 'A sentence describing your amazing service.',\n",
    "                    'ServiceOptions': {'Subset': {'VariableSubset': {'AllowMultipleValues': True}}}\n",
    "                    'SupportedProjections': ['Geographic'],\n",
    "                    'SupportedFormats': ['netCDF-4'],\n",
    "                    'Type': 'Harmony',\n",
    "                    'URL': {'Description': 'This is the harmony root endpoint',\n",
    "                            'URLValue': 'https://harmony.uat.earthdata.nasa.gov'}}\n",
    "\n",
    "service_response = requests.put(f'{base_cmr_url}/ingest/providers/{provider}/services/{service_native_id}',\n",
    "                                headers=headers,\n",
    "                                data=service_metadata)\n",
    "\n",
    "service_concept_id = json.loads(service_response).get('concept-id')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### UMM-Var records:\n",
    "\n",
    "Some services operate on an entire granule, and do not need information regarding variables. Other services may only return a user-defined selection of variables from the native granule. For this latter type of service, a Harmony request URL will include a URL-encoded full variable path. Harmony requires that UMM-Var records exist for each variable users can specify, and that these UMM-Var records are associated with the relevant collection (UMM-C) record. The service will make requests against the CMR API to retrieve the required UMM-Var records, before sending this information to the requested back-end service:\n",
    "\n",
    "There are several ways to create UMM-Var records:\n",
    "\n",
    "* Manually via the Metadata Management Tool ([MMT](https://mmt.earthdata.nasa.gov)). This is a Graphical User Interface (GUI) that is handy for creating a small number of variables and associating them with the correct collection.\n",
    "* Making HTTP requests directly against the Common Metadata Repository (CMR) API (see documentation [here](https://wiki.earthdata.nasa.gov/display/CMR/CMR+Data+Partner+User+Guide) and [here](https://cmr.earthdata.nasa.gov/ingest/site/docs/ingest/api.html#create-update-variable)).\n",
    "* Using the UMM-Var Generator ([UVG](https://uvg.earthdata.nasa.gov/)).\n",
    "\n",
    "This section will focus on using each of these methods in turn:\n",
    "\n",
    "#### Using MMT to associate variables with a collection:\n",
    "\n",
    "First navigate to the MMT instance associated with your environment (e.g. [mmt.earthdata.nasa.gov](https://mmt.earthdata.nasa.gov), [mmt.uat.earthdata.nasa.gov](mmt.uat.earthdata.nasa.gov)). Find the collection record for the collection that requires variables. On the summary page for that collection, click on the \"Create Associated Variable\" button, indicated in the figure below:\n",
    "\n",
    "![Create Associated Variable](../images/mmt_collection_page.png \"Create Associated Variable\")\n",
    "\n",
    "Clicking this link should take you to a multi-part form that allows you to fully define a UMM-Var record. Initially, the only required fields are on the first page of the form, including the variable name, long name, and definition. For variables in a hierarchical file, the variable name should be the full path, beginning with a leading \"/\" character. For variables in a flat file the leading slash is not required.\n",
    "\n",
    "After you have completed your UMM-Var record draft, you can save and publish it. The new UMM-Var record should automatically be linked to your collection.\n",
    "\n",
    "#### Making HTTP requests against CMR:\n",
    "\n",
    "This option may be preferable for a collection with a large number of variables that do not require information beyond the most basic fields (e.g., name, long name and description). API documentation is available [here](https://cmr.earthdata.nasa.gov/ingest/site/docs/ingest/api.html#create-update-variable).\n",
    "\n",
    "Requests can be made against CMR using any standard client (for example cURL). In this notebook the examples will use the Python `requests` package. This package will need to be present in your environment, and can be installed via Pip:\n",
    "\n",
    "```bash\n",
    "pip install requests\n",
    "```\n",
    "\n",
    "First you must authenticate with CMR. In the example below you will need to update the content of the XML token string with your EDL credentials, a name of your choosing for your client, and your IP address. Note, this request assumes that you are interacting with the UAT environment. If you are trying to create or update variables in another environment, you will need to update the base CMR URL near the top of this notebook:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# The echo_token variable retrieved in this cell is used in most CMR and UVG requests below.\n",
    "\n",
    "xml_token_string = ('<token><username>your_username</username>'\n",
    "                    '<password>y0ur_p4ssw0rd</password>'\n",
    "                    '<client_id>a_name_for_your_client</client_id>'\n",
    "                    '<user_ip_address>127.0.0.1</user_ip_address>'\n",
    "                    '</token>')\n",
    "\n",
    "headers = {'Content-Type': 'application/xml'}\n",
    "\n",
    "token_response = requests.post(f'{base_cmr_url}/legacy-services/rest/tokens')\n",
    "\n",
    "if token_response.status_code == 201:\n",
    "    token_response_tree = ET.from_string(token_response.content)\n",
    "    echo_token = token_response_tree.get('id').text\n",
    "    print('Successfully extracted token, ending in: ...{echo_token[-5:]}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The Echo token is a UUID contained in the \"id\" element of the response. This will need to be incorporated into the request headers of requests to create or update records in CMR.\n",
    "\n",
    "With this token in hand, one can create or update a new variable. A native ID that is unique within the collections provider must be provided. This native ID will be required every time an existing variable record is to be updated via requests to the CMR API.\n",
    "\n",
    "The cell below will create (or update) the metadata for a variable. It uses the collection concept ID defined earlier in this notebook. The content of the variable metadata dictionary is minimal. For richer examples see either the [API documentation](https://cmr.earthdata.nasa.gov/ingest/site/docs/ingest/api.html#create-update-variable) or [UMM-Var schema](https://git.earthdata.nasa.gov/projects/EMFD/repos/unified-metadata-model/browse/variable)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "headers = {'Content-type': 'application/vnd.nasa.cmr.umm+json', 'Echo-Token': echo_token}\n",
    "variable_native_id = 'sample_native_id_value'  # This must be unique to the provider.\n",
    "\n",
    "variable_metadata = {'Name': 'variable_name',\n",
    "                     'LongName': 'A long UMM-Var name',\n",
    "                     'VariableType': 'SCIENCE_VARIABLE'}\n",
    "\n",
    "var_response = requests.put(f'{base_cmr_url}/ingest/collections/{collection_concept_id}/1/variables/{variable_native_id}',\n",
    "                            headers=headers,\n",
    "                            data=variable_metadata)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The request above could be implemented as part of a script that iterates through a list of variables, if the metadata is either already known, or incredibly minimal (e.g., you already have a list of science variable names, and do not need dimension information).\n",
    "\n",
    "#### Using the UMM-Var Generator (UVG):\n",
    "\n",
    "The [UVG](http://uvg.earthdata.nasa.gov/) is powerful tool for creating UMM-Var records for collections with a large number of variables with complicated relations between one another.\n",
    "\n",
    "Documentation for UVG is available [here](https://wiki.earthdata.nasa.gov/display/UVG/UMM-Var+Generator+%28UVG%29+User%27s+Guide). First, you must use the UVG `/generate` endpoint to generate a set of valid UMM-Var records for a collection with granules in OPeNDAP. It will parse the `.dmr` for a granule (randomly selected if not specified) and return a response with valid UMM-Var records:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "headers = {'Echo-Token': echo_token}\n",
    "\n",
    "uvg_generate_response = requests.post(f'{base_uvg_url}/generate',\n",
    "                                      data={'collection_concept_id': collection_concept_id, 'provider': provider},\n",
    "                                      headers=headers)\n",
    "\n",
    "if uvg_generate_response.ok:\n",
    "    uvg_generate_json = json.loads(uvg_generate_response.content)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once valid UMM-Var records have been generated via UVG, a request can be made to publish new UMM-Var records for this collection using the `/publish` endpoint of UVG. This requires the collection concept ID, its provider, and a list of variable records, as returned in the UVG `/generate` response:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "headers = {'Echo-Token': echo_token}\n",
    "\n",
    "uvg_publish_response requests.put(f'{base_uvg_url}/publish',\n",
    "                                  data={'collection_concept_id': collection_concept_id,\n",
    "                                        'provider': provider,\n",
    "                                        'variables': uvg_generate_json.get('variables', [])}\n",
    "                                  headers=headers)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Enabling service discovery in EDSC:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After a PR has been merged to the Harmony repository that configures the service for use with Harmony, and this version of Harmony has been deployed to the necessary environment, your service will be active. At this point, however, users will not be able to discover or use this service for the relevant collections via Earthdata Search Client (EDSC). To enable this functionality, the umm-s concept that is configured for the service via the umm_s key in services.yml must be associated with the collections that can be used with the service. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Associating a UMM-S record with a collection:\n",
    "\n",
    "This can be performed either via the MMT or via the [CMR API](https://wiki.earthdata.nasa.gov/display/CMR/CMR+Data+Partner+User+Guide#CMRDataPartnerUserGuide-Services). If you are trying to add an association between a service in one provider, and a collection in another, you will have to use the CMR API.\n",
    "\n",
    "#### Associating a UMM-S record with a collection in MMT:\n",
    "\n",
    "First navigate to your UMM-S record, by searching for it in MMT. Then click on the \"Manage collection associations\" link near the top of the page:\n",
    "\n",
    "![Managing collection associations](../images/mmt_manage_associations.png \"Manage collection associations\")\n",
    "\n",
    "Within this next page you will see a list of current associations. You can the click to \"Add Collection Associations\". The page you are brought to allows you to search via fields such as collection concept ID or collection title.\n",
    "\n",
    "#### Associating a UMM-S record with a collection via the CMR API:\n",
    "\n",
    "This method is good for either associating several collections to a service in a single operation, or for associating services from other providers to a service."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# If you used MMT to create a UMM-S record, uncomment the following line, and set it to the new\n",
    "# service concept ID:\n",
    "# service_concept_id = 'S1234567890-PROVIDER'\n",
    "\n",
    "headers = {'Content-Type': 'application/json', 'Echo-Token': echo_token}\n",
    "collections_list = [{'concept_id': collection_concept_id}]\n",
    "\n",
    "association_response = requests.post(f'{base_cmr_url}/search/services/{service_concept_id}/associations',\n",
    "                                     headers=headers, data=collections_list)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Using your service via EDSC:\n",
    "\n",
    "The UMM-S record and association with a collection should take immediate effect. To test it, navigate to Earthdata Search Client (EDSC). Search for your collection and select a granule:\n",
    "\n",
    "![EDSC select a collection](../images/edsc_collection_select.png \"EDSC select a collection\")\n",
    "\n",
    "After clicking on the \"Download\" button, you'll be able to look at the download form. Near the top are the options for customizing the output. One, or more, of these options should be \"Harmony\". Select your service. If multiple Harmony services are configured for a single collection, you can choose between them by clicking on \"More Info\" to see the service name and description. Note, when a Harmony request is received for a collection with multiple services, Harmony will try to route the request to the service chain that can best fulfill the request, accounting for the input request parameters.\n",
    "\n",
    "![EDSC customise download](../images/edsc_download_form_one.png \"EDSC customise download\")\n",
    "\n",
    "After selecting your Harmony service, the download form should include all the options to provide data that your service needs. In the example below, a user can request only a subset of the variables to be returned from the original input granule. Once the form is complete, you can then click \"Download Data\" and you will be redirected to the standard status page for an EDSC download request.\n",
    "\n",
    "![EDSC customise download](../images/edsc_download_form_two.png \"EDSC customise download\")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}