CUMULUS-3694: Update granules List endpoints to query postgres - filter by field value #3656

jennyhliu · 2024-05-10T20:38:29Z

Summary: Summary of changes

Addresses CUMULUS-3694: Update granules List endpoints to query postgres - filter by field value

Changes

Added functionality to @cumulus/db/src/search to support term queries
Updated BaseSearch and GranuleSearch classes to support term queries for granules
Updated granules List endpoint to search postgres

PR Checklist

Update CHANGELOG
Unit tests
Ad-hoc testing - Deploy changes and test manually
Integration tests

…o jl/CUMULUS-3694

Nnaga1 · 2024-05-13T17:32:38Z

packages/api/endpoints/granules.js

-  } else {
-    result = await es.query();
-  }
+  const dbSearch = new GranuleSearch({ queryStringParameters });


Nnaga1 · 2024-05-13T17:48:35Z

packages/api/tests/endpoints/test-granules.js

+    .set('Accept', 'application/json')
+    .set('Authorization', `Bearer ${jwtAuthToken}`)
+    .expect(200);
+


I don't really understand this test, when you add granuleId: granuleIds[3], to the SearchParams, why is the meta count 2? shouldn't 1 granuleid = 1 distinct granule, maybe I'm missing something, I get the t.is(results.length, 1); because 1 granule is returned

There are 2 granules with same granule Id but different collections in the setup

ohhhhhh ok, I didn't know it could work like that, thanks 👍

Nnaga1 · 2024-05-13T17:50:20Z

packages/db/src/search/BaseSearch.ts

-    this.dbQueryParameters.limit = Number.parseInt(
-      (this.queryStringParameters.limit) ?? '10',
-      10
+    this.dbQueryParameters = convertQueryStringToDbQueryParameters(


Nnaga1 · 2024-05-13T17:52:42Z

packages/db/src/search/BaseSearch.ts

@@ -51,27 +51,32 @@ class BaseSearch {
      searchQuery: Knex.QueryBuilder,
    } {
    const { countQuery, searchQuery } = this.buildBasicQuery(knex);
-    if (this.dbQueryParameters.limit) searchQuery.limit(this.dbQueryParameters.limit);
-    if (this.dbQueryParameters.offset) searchQuery.offset(this.dbQueryParameters.offset);
+    this.buildTermQuery({ countQuery, searchQuery });


so for here, you need to specify 2 different types of queries in case the query might be a term one or an infix one? I don't quite understand how that works, or do both queries (count and search) need to be executed and parsed for the api response

Unlike es, the search query doesn't return count, and only returns limited number of records, so we need a separate query to get the total count which satisfy the search parameters.

A search can have infix, term and other supported query types (sort, range etc.)

yea I understood this after looking at your GranuleSearch class, it was kinda hard to understand how you did it based on my initial understanding of the stats endpoint (since stats mainly does count, aggregation, avg, not actually returning records), I understand now the meta thing needs a count and then the actual body is the granules records actually returned 👍

Nnaga1 · 2024-05-13T17:53:11Z

packages/db/src/search/BaseSearch.ts

    return { countQuery, searchQuery };
  }

  /**
-   * metadata template for query result
+   * Get metadata template for query result
   *
   * @returns metadata template
   */
  private _metaTemplate(): Meta {


👍 nice, this is something I can use

Nnaga1 · 2024-05-13T18:14:20Z

packages/db/src/search/GranuleSearch.ts

+  protected buildInfixPrefixQuery(params: {
+    countQuery: Knex.QueryBuilder,
+    searchQuery: Knex.QueryBuilder,
+    dbQueryParameters?: DbQueryParameters,


this is great 🙌

Nnaga1 · 2024-05-13T18:15:01Z

packages/db/src/search/GranuleSearch.ts

+        searchQuery.where(`${pdrsTable}.name`, value);
+      }
+      if (name === 'error.Error') {
+        countQuery.whereRaw(`${granulesTable}.error->>'Error' = '${value}'`);


awesome!! 👍

Nnaga1 · 2024-05-13T18:16:49Z

packages/db/src/search/field-mapping.ts

+const log = new Logger({ sender: '@cumulus/db/field-mapping' });
+
+// functions to map the api search string field name and value to postgres db field
+const granuleMapping: { [key: string]: Function } = {


looks good!

Nnaga1 · 2024-05-13T18:19:48Z

packages/db/src/search/queries.ts

+ * @param queryStringParameters - query string parameters
+ * @returns db query parameters
+ */
+export const convertQueryStringToDbQueryParameters = (


a lot of stuff here seems helpful for the queries around granules, collections, executions, providers, etc. not really stats since there is no stats table

Nnaga1 · 2024-05-13T18:22:47Z

packages/db/tests/search/test-GranuleSearch.js

+    fields,
+  };
+  const dbSearch = new GranuleSearch({ queryStringParameters });
+  const response = await dbSearch.query(knex);


these tests look great, maybe adding one where it queries by collection and provider would be helpful?

Only GranuleSearch is implemented at this moment.

yea but like this:

get granules?collectionId=testCollection1&providerId=testProvider1, would that be useful or neccessary in your case?

Nnaga1

Thanks for adding in the tests and answering my questions, LGTM 🙌

…er by field value (#3656) * CUMULUS-3692:Granule list endpoint for basic postgres query * refactor * refactor * typing * add changelog entry * skip search_after * skip searchafter unit tests * add granule list test * rename * refactor * build query parameters * update comment * add field-mapping * update jsdoc * use type over interface,add log * update test description * build term/terms * buildDbQueryParameters * add unit test no terms search * add doc * rename * add unit test * add fields test * add more unit tests * support error.Error search * fix lint * rename functions * ignore files * add convert query unit tests * add all types * add unit test for fieldmapping types fix timestamp * update timestamp test * add multiple term field test * ignore execution in granule list record

* CUMULUS-3692: Update granules List endpoints to query postgres for basic queries (#3637) * CUMULUS-3692:Granule list endpoint for basic postgres query * CUMULUS-3694: Update granules List endpoints to query postgres - filter by field value (#3656) * CUMULUS-3692:Granule list endpoint for basic postgres query * refactor * refactor * typing * add changelog entry * skip search_after * skip searchafter unit tests * add granule list test * rename * refactor * build query parameters * update comment * add field-mapping * update jsdoc * use type over interface,add log * update test description * build term/terms * buildDbQueryParameters * add unit test no terms search * add doc * rename * add unit test * add fields test * add more unit tests * support error.Error search * fix lint * rename functions * ignore files * add convert query unit tests * add all types * add unit test for fieldmapping types fix timestamp * update timestamp test * add multiple term field test * ignore execution in granule list record * CUMULUS-3689: Update Stats/Summary and Stats/Aggregate endpoints to use psql (#3659) * first commit on new branch * CHANGELOG change * small fix * PR feedback * adding jsdoc + fixing spelling/grammar * CUMULUS-3693: Update granules List endpoints to query postgres - range (#3660) * add range query support * CUMULUS-3695 - Update Granules endpoint to handle SortFields (#3663) * first committ * CHANGELOG change * fixing sortQueryMethod * simplifying code * PR feedback * merge conflicts + improving code * small jsdoc fix * PR feedback * PR feedback * PR feedback * fixing test * PR feedback * PR feedback * CUMULUS-3696: Update granules List endpoints to query postgres - match (#3674) * add methods to convert terms,not,exists * CUMULUS-3641 - Update Collections LIST endpoint to query Postgres basic (#3681) * reopening PR * PR feedback * small test fix * small PR feedbacks * adding new tests from match queries * PR feedback/formatting * temporary reversion to list endpoint for reconreport tests * reverting changes * adding logging * more logging * more logging * removing logging + commenting reconrep test temp * commenting out failing createReconReport spec * removing comment * reverting changes to reconReport test * reverting previous change * adding ts-check * PR feedback * PR feedback * adding in test * PR feedback fix * PR feedback * CUMULUS-3699 - Update collection List endpoints to query postgres - includeStats (#3688) * first commit * CHANGELOG * fixing small things * changes + fixes * PR feedback * splitting queries separately * PR feedback * PR feedback * PR feedback * CUMULUS-3639: Add support to db/CollectionSearch to retrieve active collections (#3693) * CUMULUS-3639:Add support to db/CollectionSearch to retrieve active collections * add test active collections * add ts-check * update /collections/active unit test * test snyk * fix field mapping * parallel search and fix urlPath * add cumulus-lp stack * add limit 1 to subquery * CUMULUS-3239 - Update Executions LIST endpoint to query Postgres basic (#3684) * first commit * adding changes * storing changes * updating progress * linting + small fixes * small fix * changing timestamp to string in tests * fixing timestamp * commenting out tests failing in CI but not locally * saving changes * collection support * adding async_ops support * changing endpoint + tests * fixing test * uncommenting tests + adding var * commenting out tests failing in CI but not locally * adding parentArn support + changing tests * added parentArn support + fixing tests * small endpoint test fix * Pr feedback + code improvements * small CHANGELOG fix * PR feedback * PR feedback + linting * PR feedback * PR feedback * fixing test * fixing execution tests after removing asyncCumulusOPId from mapping * PR feedback * removed includeFullRecord from search classes * PR feedback * PR feedback * reverting change * fix changelog * Cumulus 3640/3242- Update granule non-LIST endpoints and other granule related es queries to query postgres (#3727) * removing granules * fixing lint * fixing test * small change * adding back in some deleted things * removing more * lint fix * removing tests * skipping execution search-by-granules tests * skipping execution tests * removing tests * more removing * adding in deleted test * removing more * adding back in needed code * removing ES_HOST, query, and index from bulk_ops * fixing bulk_ops tests * adding back in ELK stack refs * changing reconreports test to skip/adding back in getGranulesByPayload * PR feedback * adding back in skipped tests * CHANGELOG * PR feedback * PR feedback' . ; * PR feedback + syntax check * adding back sort to write-granules test * CUMULUS-3240: Remove Elasticsearch dependency from executions endpoints (#3723) * CUMULUS-3240:Remove ElasticSearch dependency from Executions endpoints * update test-executions * remove es dependencies for execution * update changelog * fix lint and warning * address PR feedback * remove esClient from createExecutionRecords * CUMULUS-3642: postgres query adjustment (#3731) * estimate table row count * add more indexes * use count * * fix test * fix test * fix lint and add file index * add more index * fix lint * fix lint update test * execution asyncOperationId is optional * defautl stats last day * fix granule patchByGranuleId put logic back * vaccum tables * update changelog * remove sql script to another pr * update changelog * update active collection query * Revert "update active collection query" This reverts commit 88024c2. * CUMULUS-3238: Remove ElasticSearch dependency from Collection POST, PUT, and DEL endpoints (#3746) * first commit * CHANGELOG change * PR feedback * PR feedback * CUMULUS-3792: Add db indexes to improve search performance (#3751) * CUMULUS-3792:Add table indexes to improve search performance * fix changelog --------- Co-authored-by: Naga Nages <66387215+Nnaga1@users.noreply.github.com> Co-authored-by: Nate Pauzenga <npauzenga@gmail.com>

jennyhliu and others added 29 commits April 24, 2024 14:19

CUMULUS-3692:Granule list endpoint for basic postgres query

0772a4e

refactor

dfdf68f

refactor

4efe990

Merge branch 'master' into jl/CUMULUS-3692

38f588d

typing

5cc833c

add changelog entry

883e65f

skip search_after

cd20bd4

skip searchafter unit tests

08d635a

Merge branch 'master' into jl/CUMULUS-3692

020ccb6

add granule list test

e3926c7

rename

935641d

Merge branch 'master' into jl/CUMULUS-3692

8d971e9

refactor

c1e724a

build query parameters

c31a9d9

update comment

421a2dd

add field-mapping

87b3cdf

update jsdoc

9942fac

use type over interface,add log

39922be

update test description

fb9c51f

Merge branch 'jl/CUMULUS-3692' into jl/CUMULUS-3694

0f36350

Merge branch 'jl/CUMULUS-3692' into jl/CUMULUS-3694

a655b7d

build term/terms

facd22f

buildDbQueryParameters

8eeb5e2

add unit test no terms search

ae0e085

add doc

a68a67a

rename

73504c4

add unit test

e571757

add fields test

2610483

add more unit tests

31c1ec3

jennyhliu added the work in progress label May 10, 2024

jennyhliu changed the base branch from master to feature/es-phase-1 May 10, 2024 20:40

jennyhliu and others added 8 commits May 10, 2024 16:46

Merge branch 'feature/es-phase-1' into jl/CUMULUS-3694

c36a7b4

support error.Error search

a1a58f7

Merge branch 'jl/CUMULUS-3694' of https://github.com/nasa/cumulus int…

7ce65a7

…o jl/CUMULUS-3694

fix lint

cef467b

rename functions

e6339db

ignore files

e899aca

add convert query unit tests

b66a580

add all types

ee593d1

Nnaga1 reviewed May 13, 2024

View reviewed changes

jennyhliu added 2 commits May 13, 2024 15:23

add unit test for fieldmapping types fix timestamp

c8ae699

update timestamp test

b4e56b0

jennyhliu marked this pull request as ready for review May 13, 2024 19:30

jennyhliu added in review and removed work in progress labels May 13, 2024

jennyhliu added 2 commits May 13, 2024 16:46

add multiple term field test

d522b24

ignore execution in granule list record

470f354

Nnaga1 approved these changes May 13, 2024

View reviewed changes

jennyhliu merged commit 1c2a666 into feature/es-phase-1 May 13, 2024
3 checks passed

jennyhliu deleted the jl/CUMULUS-3694 branch May 13, 2024 23:44

jennyhliu removed in review labels May 13, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CUMULUS-3694: Update granules List endpoints to query postgres - filter by field value #3656

CUMULUS-3694: Update granules List endpoints to query postgres - filter by field value #3656

jennyhliu commented May 10, 2024 •

edited

Loading

Nnaga1 May 13, 2024

Nnaga1 May 13, 2024

jennyhliu May 13, 2024

Nnaga1 May 13, 2024

Nnaga1 May 13, 2024

Nnaga1 May 13, 2024

jennyhliu May 13, 2024

jennyhliu May 13, 2024

Nnaga1 May 13, 2024 •

edited

Loading

Nnaga1 May 13, 2024

Nnaga1 May 13, 2024

Nnaga1 May 13, 2024

Nnaga1 May 13, 2024

Nnaga1 May 13, 2024

Nnaga1 May 13, 2024

jennyhliu May 13, 2024

Nnaga1 May 13, 2024

Nnaga1 left a comment •

edited

Loading

CUMULUS-3694: Update granules List endpoints to query postgres - filter by field value #3656

CUMULUS-3694: Update granules List endpoints to query postgres - filter by field value #3656

Conversation

jennyhliu commented May 10, 2024 • edited Loading

Changes

PR Checklist

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Nnaga1 May 13, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Nnaga1 left a comment • edited Loading

Choose a reason for hiding this comment

jennyhliu commented May 10, 2024 •

edited

Loading

Nnaga1 May 13, 2024 •

edited

Loading

Nnaga1 left a comment •

edited

Loading