Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

CUMULUS-3692: Update granules List endpoints to query postgres for basic queries #3637

Merged
merged 17 commits into from
May 8, 2024
Merged

CUMULUS-3692: Update granules List endpoints to query postgres for basic queries #3637

merged 17 commits into from
May 8, 2024

Conversation

jennyhliu
Copy link
Contributor

@jennyhliu jennyhliu commented Apr 29, 2024 •
edited
Loading

Summary: Summary of changes

Addresses CUMULUS-3692: Update granules List endpoints to query postgres - basic

Changes

  • Update granules List endpoints to query postgres for basic queries

Note:

  • Skip searchAfter/searchContext unit tests since they are not supported by postgres. We should enable/update for CUMULUS-3697
  • We will create a feature branch and merge the ticket to that until all granule list endpoints are in place and performance tested.

PR Checklist

  • Update CHANGELOG
  • Unit tests
  • Ad-hoc testing - Deploy changes and test manually
  • Integration tests

@jennyhliu jennyhliu marked this pull request as ready for review April 29, 2024 22:54
charleshuang80
charleshuang80 reviewed May 6, 2024
View reviewed changes
Copy link
Contributor

@charleshuang80 charleshuang80 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall looks good, a few minor questions and comments.

packages/api/endpoints/granules.js Show resolved Hide resolved
packages/api/tests/endpoints/granules/test-searchafter-10k.js Show resolved Hide resolved
packages/db/src/translate/granules.ts Show resolved Hide resolved
packages/db/src/translate/granules.ts Show resolved Hide resolved
packages/api/tests/endpoints/test-granules.js Outdated Show resolved Hide resolved
@jennyhliu jennyhliu changed the base branch from master to feature/es-phase-1 May 6, 2024 19:21
@jennyhliu jennyhliu merged commit 81018b2 into feature/es-phase-1 May 8, 2024
3 checks passed
jennyhliu added a commit that referenced this pull request Aug 27, 2024
@jennyhliu @Nnaga1 @npauzenga
feature/es-phase-1 (#3648)
04845d7 
* CUMULUS-3692: Update granules List endpoints to query postgres for basic queries (#3637)

* CUMULUS-3692:Granule list endpoint for basic postgres query

* CUMULUS-3694: Update granules List endpoints to query postgres - filter by field value (#3656)

* CUMULUS-3692:Granule list endpoint for basic postgres query

* refactor

* refactor

* typing

* add changelog entry

* skip search_after

* skip searchafter unit tests

* add granule list test

* rename

* refactor

* build query parameters

* update comment

* add field-mapping

* update jsdoc

* use type over interface,add log

* update test description

* build term/terms

* buildDbQueryParameters

* add unit test no terms search

* add doc

* rename

* add unit test

* add fields test

* add more unit tests

* support error.Error search

* fix lint

* rename functions

* ignore files

* add convert query unit tests

* add all types

* add unit test for fieldmapping types fix timestamp

* update timestamp test

* add multiple term field test

* ignore execution in granule list record

* CUMULUS-3689: Update Stats/Summary and Stats/Aggregate endpoints to use psql (#3659)

* first commit on new branch

* CHANGELOG change

* small fix

* PR feedback

* adding jsdoc + fixing spelling/grammar

* CUMULUS-3693: Update granules List endpoints to query postgres - range (#3660)

* add range query support

* CUMULUS-3695 - Update Granules endpoint to handle SortFields (#3663)

* first committ

* CHANGELOG change

* fixing sortQueryMethod

* simplifying code

* PR feedback

* merge conflicts + improving code

* small jsdoc fix

* PR feedback

* PR feedback

* PR feedback

* fixing test

* PR feedback

* PR feedback

* CUMULUS-3696: Update granules List endpoints to query postgres - match (#3674)

* add methods to convert terms,not,exists

* CUMULUS-3641 - Update Collections LIST endpoint to query Postgres basic  (#3681)

* reopening PR

* PR feedback

* small test fix

* small PR feedbacks

* adding new tests from match queries

* PR feedback/formatting

* temporary reversion to list endpoint for reconreport tests

* reverting changes

* adding logging

* more logging

* more logging

* removing logging + commenting reconrep test temp

* commenting out failing createReconReport spec

* removing comment

* reverting changes to reconReport test

* reverting previous change

* adding ts-check

* PR feedback

* PR feedback

* adding in test

* PR feedback fix

* PR feedback

* CUMULUS-3699 - Update collection List endpoints to query postgres - includeStats (#3688)

* first commit

* CHANGELOG

* fixing small things

* changes + fixes

* PR feedback

* splitting queries separately

* PR feedback

* PR feedback

* PR feedback

* CUMULUS-3639: Add support to db/CollectionSearch to retrieve active collections (#3693)

* CUMULUS-3639:Add support to db/CollectionSearch to retrieve active collections

* add test active collections

* add ts-check

* update /collections/active unit test

* test snyk

* fix field mapping

* parallel search and fix urlPath

* add cumulus-lp stack

* add limit 1 to subquery

* CUMULUS-3239 - Update Executions LIST endpoint to query Postgres basic (#3684)

* first commit

* adding changes

* storing changes

* updating progress

* linting + small fixes

* small fix

* changing timestamp to string in tests

* fixing timestamp

* commenting out tests failing in CI but not locally

* saving changes

* collection support

* adding async_ops support

* changing endpoint + tests

* fixing test

* uncommenting tests + adding var

* commenting out tests failing in CI but not locally

* adding parentArn support + changing tests

* added parentArn support + fixing tests

* small endpoint test fix

* Pr feedback + code improvements

* small CHANGELOG fix

* PR feedback

* PR feedback + linting

* PR feedback

* PR feedback

* fixing test

* fixing execution tests after removing asyncCumulusOPId from mapping

* PR feedback

* removed includeFullRecord from search classes

* PR feedback

* PR feedback

* reverting change

* fix changelog

* Cumulus 3640/3242- Update granule non-LIST endpoints and other granule related es queries to query postgres (#3727)

* removing granules

* fixing lint

* fixing test

* small change

* adding back in some deleted things

* removing more

* lint fix

* removing tests

* skipping execution search-by-granules tests

* skipping execution tests

* removing tests

* more removing

* adding in deleted test

* removing more

* adding back in needed code

* removing ES_HOST, query, and index from bulk_ops

* fixing bulk_ops tests

* adding back in ELK stack refs

* changing reconreports test to skip/adding back in getGranulesByPayload

* PR feedback

* adding back in skipped tests

* CHANGELOG

* PR feedback

* PR feedback' .
;

* PR feedback + syntax check

* adding back sort to write-granules test

* CUMULUS-3240: Remove Elasticsearch dependency from executions endpoints (#3723)

* CUMULUS-3240:Remove ElasticSearch dependency from Executions endpoints

* update test-executions

* remove es dependencies for execution

* update changelog

* fix lint and warning

* address PR feedback

* remove esClient from createExecutionRecords

* CUMULUS-3642: postgres query adjustment (#3731)

* estimate table row count

* add more indexes

* use count *

* fix test

* fix test

* fix lint and add file index

* add more index

* fix lint

* fix lint update test

* execution asyncOperationId is optional

* defautl stats last day

* fix granule patchByGranuleId put logic back

* vaccum tables

* update changelog

* remove sql script to another pr

* update changelog

* update active collection query

* Revert "update active collection query"

This reverts commit 88024c2.

* CUMULUS-3238: Remove ElasticSearch dependency from Collection POST, PUT, and DEL endpoints (#3746)

* first commit

* CHANGELOG change

* PR feedback

* PR feedback

* CUMULUS-3792: Add db indexes to improve search performance (#3751)

* CUMULUS-3792:Add table indexes to improve search performance

* fix changelog

---------

Co-authored-by: Naga Nages <66387215+Nnaga1@users.noreply.github.com>
Co-authored-by: Nate Pauzenga <npauzenga@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@charleshuang80 charleshuang80 charleshuang80 approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@jennyhliu @charleshuang80