CUMULUS-3693: Update granules List endpoints to query postgres - range #3660

jennyhliu · 2024-05-17T17:47:09Z

Summary: Summary of changes

Addresses CUMULUS-3693: Update granules List endpoints to query postgres - range

Changes

Added functionality to @cumulus/db/src/search to support range queries

PR Checklist

Update CHANGELOG
Unit tests
Ad-hoc testing - Deploy changes and test manually
Integration tests

Jkovarik

This change does not fix the problem - we should not be using promisify.

Nnaga1 · 2024-05-22T15:51:22Z

example/config.yml

@@ -8,6 +8,11 @@ cumulus-sit:
  apiUsername: jasmine
  pdrNodeNameProviderBucket: cumulus-sit-pdr-node-name-provider

+cumulus-es:


so for this, should PRs run on this branch use this stack from now on you think?

Nnaga1 · 2024-05-22T15:59:29Z

packages/db/src/search/BaseSearch.ts

@@ -33,18 +33,33 @@ export const typeToTable: { [key: string]: string } = {
 */
 class BaseSearch {
  readonly type: string;
+  readonly tableName: string;


are you adding this to the BaseSearch just for ease/making the code cleaner ? same with the searchCollection/Provider/Pdr,

Nnaga1 · 2024-05-22T16:02:04Z

packages/db/src/search/BaseSearch.ts

+    const { countQuery, searchQuery, dbQueryParameters } = params;
+    const { range = {} } = dbQueryParameters ?? this.dbQueryParameters;
+
+    Object.entries(range).forEach(([name, rangeValues]) => {


so range is like

timestamp_to: { lte: 11/1/11, gte: 11/2/12} ? and this deconstructs and queries like that, makes sense 👍

Nnaga1 · 2024-05-22T16:03:05Z

packages/db/src/search/BaseSearch.ts


    Object.entries(term).forEach(([name, value]) => {
-      countQuery?.where(`${table}.${name}`, value);
-      searchQuery.where(`${table}.${name}`, value);
+      switch (name) {


you are adding this here to reduce code redundancy across the other search classes that extend base search right?

Nnaga1 · 2024-05-22T16:03:47Z

packages/db/src/search/GranuleSearch.ts

@@ -36,21 +34,6 @@ export class GranuleSearch extends BaseSearch {
    super(event, 'granule');
  }

-  private searchCollection(): boolean {
-    const term = this.dbQueryParameters.term;


Nnaga1 · 2024-05-22T16:11:04Z

packages/db/src/search/field-mapping.ts

@@ -138,6 +138,14 @@ const executionMapping : { [key: string]: Function } = {
  updatedAt: (value?: string) => ({
    updated_at: value && new Date(Number(value)),
  }),
+  // The following fields require querying other tables
+  collectionId: (value?: string) => {


did the range code-changes you did need this for you to add or was this just refactoring

Nnaga1 · 2024-05-22T16:16:15Z

packages/db/src/search/queries.ts

+  type: string,
+  queryStringFields: { name: string, value: string }[]
+): { range: { [key: string]: RangeType } } => {
+  const range = queryStringFields.reduce((acc: { [key: string]: RangeType }, queryField) => {


I'm a little confused by this, can you explain what this does/how it works? so if I have timestamp__from and timestamp__to for the range, what would this do? make it into the field: { gte: x, lte: y) ?

Nnaga1 · 2024-05-22T16:19:58Z

packages/db/tests/search/test-GranuleSearch.js

+  t.is(response.results?.length, 100);
+
+  queryStringParameters = {
+    limit: 200,


awesome 🙌

Nnaga1 · 2024-05-22T16:20:50Z

packages/db/tests/search/test-StatsSearch.js

@@ -196,7 +191,6 @@ test('StatsSearch filters executions correctly', async (t) => {
  t.deepEqual(results2.count, expectedResponse2);

  queryStringParameters = {
-    type: 'executions',


thanks 👍

Nnaga1 · 2024-05-22T16:21:54Z

packages/db/src/search/StatsSearch.ts

-      query.join(`${TableNames.collections}`, `${this.tableName}.collection_cumulus_id`, 'collections.cumulus_id');
+    const {
+      collections: collectionsTable,
+      providers: providersTable,


thanks 👍

Nnaga1

Discussed changes in huddle, great work 🚀

* CUMULUS-3692: Update granules List endpoints to query postgres for basic queries (#3637) * CUMULUS-3692:Granule list endpoint for basic postgres query * CUMULUS-3694: Update granules List endpoints to query postgres - filter by field value (#3656) * CUMULUS-3692:Granule list endpoint for basic postgres query * refactor * refactor * typing * add changelog entry * skip search_after * skip searchafter unit tests * add granule list test * rename * refactor * build query parameters * update comment * add field-mapping * update jsdoc * use type over interface,add log * update test description * build term/terms * buildDbQueryParameters * add unit test no terms search * add doc * rename * add unit test * add fields test * add more unit tests * support error.Error search * fix lint * rename functions * ignore files * add convert query unit tests * add all types * add unit test for fieldmapping types fix timestamp * update timestamp test * add multiple term field test * ignore execution in granule list record * CUMULUS-3689: Update Stats/Summary and Stats/Aggregate endpoints to use psql (#3659) * first commit on new branch * CHANGELOG change * small fix * PR feedback * adding jsdoc + fixing spelling/grammar * CUMULUS-3693: Update granules List endpoints to query postgres - range (#3660) * add range query support * CUMULUS-3695 - Update Granules endpoint to handle SortFields (#3663) * first committ * CHANGELOG change * fixing sortQueryMethod * simplifying code * PR feedback * merge conflicts + improving code * small jsdoc fix * PR feedback * PR feedback * PR feedback * fixing test * PR feedback * PR feedback * CUMULUS-3696: Update granules List endpoints to query postgres - match (#3674) * add methods to convert terms,not,exists * CUMULUS-3641 - Update Collections LIST endpoint to query Postgres basic (#3681) * reopening PR * PR feedback * small test fix * small PR feedbacks * adding new tests from match queries * PR feedback/formatting * temporary reversion to list endpoint for reconreport tests * reverting changes * adding logging * more logging * more logging * removing logging + commenting reconrep test temp * commenting out failing createReconReport spec * removing comment * reverting changes to reconReport test * reverting previous change * adding ts-check * PR feedback * PR feedback * adding in test * PR feedback fix * PR feedback * CUMULUS-3699 - Update collection List endpoints to query postgres - includeStats (#3688) * first commit * CHANGELOG * fixing small things * changes + fixes * PR feedback * splitting queries separately * PR feedback * PR feedback * PR feedback * CUMULUS-3639: Add support to db/CollectionSearch to retrieve active collections (#3693) * CUMULUS-3639:Add support to db/CollectionSearch to retrieve active collections * add test active collections * add ts-check * update /collections/active unit test * test snyk * fix field mapping * parallel search and fix urlPath * add cumulus-lp stack * add limit 1 to subquery * CUMULUS-3239 - Update Executions LIST endpoint to query Postgres basic (#3684) * first commit * adding changes * storing changes * updating progress * linting + small fixes * small fix * changing timestamp to string in tests * fixing timestamp * commenting out tests failing in CI but not locally * saving changes * collection support * adding async_ops support * changing endpoint + tests * fixing test * uncommenting tests + adding var * commenting out tests failing in CI but not locally * adding parentArn support + changing tests * added parentArn support + fixing tests * small endpoint test fix * Pr feedback + code improvements * small CHANGELOG fix * PR feedback * PR feedback + linting * PR feedback * PR feedback * fixing test * fixing execution tests after removing asyncCumulusOPId from mapping * PR feedback * removed includeFullRecord from search classes * PR feedback * PR feedback * reverting change * fix changelog * Cumulus 3640/3242- Update granule non-LIST endpoints and other granule related es queries to query postgres (#3727) * removing granules * fixing lint * fixing test * small change * adding back in some deleted things * removing more * lint fix * removing tests * skipping execution search-by-granules tests * skipping execution tests * removing tests * more removing * adding in deleted test * removing more * adding back in needed code * removing ES_HOST, query, and index from bulk_ops * fixing bulk_ops tests * adding back in ELK stack refs * changing reconreports test to skip/adding back in getGranulesByPayload * PR feedback * adding back in skipped tests * CHANGELOG * PR feedback * PR feedback' . ; * PR feedback + syntax check * adding back sort to write-granules test * CUMULUS-3240: Remove Elasticsearch dependency from executions endpoints (#3723) * CUMULUS-3240:Remove ElasticSearch dependency from Executions endpoints * update test-executions * remove es dependencies for execution * update changelog * fix lint and warning * address PR feedback * remove esClient from createExecutionRecords * CUMULUS-3642: postgres query adjustment (#3731) * estimate table row count * add more indexes * use count * * fix test * fix test * fix lint and add file index * add more index * fix lint * fix lint update test * execution asyncOperationId is optional * defautl stats last day * fix granule patchByGranuleId put logic back * vaccum tables * update changelog * remove sql script to another pr * update changelog * update active collection query * Revert "update active collection query" This reverts commit 88024c2. * CUMULUS-3238: Remove ElasticSearch dependency from Collection POST, PUT, and DEL endpoints (#3746) * first commit * CHANGELOG change * PR feedback * PR feedback * CUMULUS-3792: Add db indexes to improve search performance (#3751) * CUMULUS-3792:Add table indexes to improve search performance * fix changelog --------- Co-authored-by: Naga Nages <66387215+Nnaga1@users.noreply.github.com> Co-authored-by: Nate Pauzenga <npauzenga@gmail.com>

jennyhliu added 2 commits May 16, 2024 11:06

add range query support

10a7d3b

add range search test

2b9a58c

jennyhliu added the work in progress label May 17, 2024

jennyhliu and others added 8 commits May 17, 2024 13:48

Merge branch 'feature/es-phase-1' into jl/CUMULUS-3693

a9eb2e2

merge conflict

0776c5e

add range to StatsSearch

ba29267

refactor collectionId to name version

8183b51

move collectionId search to base

91e86ed

refactor

46f1a75

update changelog

6d40fbe

fix HttpProviderClient ci

3399130

Jkovarik requested changes May 21, 2024

View reviewed changes

jennyhliu added 2 commits May 21, 2024 11:24

remove pipeline fix

c23eaac

remove pipeline fix

a1ecf50

jennyhliu marked this pull request as ready for review May 21, 2024 16:10

Merge branch 'feature/es-phase-1' into jl/CUMULUS-3693

241f361

jennyhliu added needs review and removed work in progress labels May 22, 2024

Nnaga1 added in review and removed needs review labels May 22, 2024

Nnaga1 reviewed May 22, 2024

View reviewed changes

jennyhliu added 2 commits May 22, 2024 15:56

cmr launchpad ingest

88b3136

add comments

3800bcd

Nnaga1 approved these changes May 23, 2024

View reviewed changes

jennyhliu added 2 commits May 23, 2024 15:17

debug integration error

441ccbb

remove logging

bd11d23

jennyhliu removed the in review label May 24, 2024

jennyhliu merged commit ad28841 into feature/es-phase-1 May 25, 2024
3 checks passed

jennyhliu deleted the jl/CUMULUS-3693 branch May 25, 2024 00:59

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CUMULUS-3693: Update granules List endpoints to query postgres - range #3660

CUMULUS-3693: Update granules List endpoints to query postgres - range #3660

jennyhliu commented May 17, 2024 •

edited

Loading

Jkovarik left a comment

Nnaga1 May 22, 2024

Nnaga1 May 22, 2024

Nnaga1 May 22, 2024

Nnaga1 May 22, 2024

Nnaga1 May 22, 2024

Nnaga1 May 22, 2024 •

edited

Loading

Nnaga1 May 22, 2024

Nnaga1 May 22, 2024

Nnaga1 May 22, 2024

Nnaga1 May 22, 2024

Nnaga1 left a comment

CUMULUS-3693: Update granules List endpoints to query postgres - range #3660

CUMULUS-3693: Update granules List endpoints to query postgres - range #3660

Conversation

jennyhliu commented May 17, 2024 • edited Loading

Changes

PR Checklist

Jkovarik left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Nnaga1 May 22, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Nnaga1 left a comment

Choose a reason for hiding this comment

jennyhliu commented May 17, 2024 •

edited

Loading

Nnaga1 May 22, 2024 •

edited

Loading