Subscriber and Dowload warns not all granules will be downloaded, but all granules get downloaded #99
Labels
bug
Something isn't working
Comments
mike-gangl
added a commit
that referenced
this issue
Aug 18, 2022
Merged
mike-gangl
added a commit
that referenced
this issue
Aug 23, 2022
mike-gangl
added a commit
that referenced
this issue
Dec 12, 2022
* Develop (#103) * Issues/91 (#92) * added citation creation tests and functionality to subscriber and downloader * added verbose option to create_citation_file command, previously hard coded * updated changelog (whoops) and fixed regression test: 1. Issue where the citation file now downloaded affected the counts 2. Issue where the logic for determining if a file modified time was changing or not was picking up the new citation file which _always_ gets rewritten to update the 'last accessed' date. * updated request to include exec_info in warning; fixed issue with params not being a dictionary caused errors * changed a warning to debug for citation file. fixed test issues * Enable debug logging during regression tests and set max parallel workflows to 2 * added output to pytest * fixed test to only look for downlaoded data files not citation file due to 'random' cmr errors when creating a citation. * added mock testing and retry on 503 * added 503 fixes Co-authored-by: Frank Greguska <Francis.Greguska@jpl.nasa.gov> * fixed issues where token was not proagated to CMR queries (#95) * Misc fixes (#101) * added ".tiff" to default extensions to address #100 * removed 'warning' message on not downloading all data to close #99 * updated help documentation for start/end times to close #79 * added version update, updates to CHANGELOG * added token get,delete, refresh and list operations * Revert "added token get,delete, refresh and list operations" This reverts commit 15aba90. * Update python-app.yml Co-authored-by: Frank Greguska <Francis.Greguska@jpl.nasa.gov> * updated poetry version Version matches build/test versions. * Update README.md * Update podaac_data_downloader.py Fixing for issues 109 - adding capability to download by granule-name * Update Downloader.md Fixed the help file * added changelog entries, regressiont ests * added poetry lock cleanup Co-authored-by: Frank Greguska <Francis.Greguska@jpl.nasa.gov> Co-authored-by: Frank Greguska <89428916+frankinspace@users.noreply.github.com> Co-authored-by: sureshshsv <45676320+sureshshsv@users.noreply.github.com> Co-authored-by: sureshshsv <suresh.vannan@jpl.nasa.gov>
mike-gangl
added a commit
that referenced
this issue
Dec 12, 2022
* Issues/91 (#92) * added citation creation tests and functionality to subscriber and downloader * added verbose option to create_citation_file command, previously hard coded * updated changelog (whoops) and fixed regression test: 1. Issue where the citation file now downloaded affected the counts 2. Issue where the logic for determining if a file modified time was changing or not was picking up the new citation file which _always_ gets rewritten to update the 'last accessed' date. * updated request to include exec_info in warning; fixed issue with params not being a dictionary caused errors * changed a warning to debug for citation file. fixed test issues * Enable debug logging during regression tests and set max parallel workflows to 2 * added output to pytest * fixed test to only look for downlaoded data files not citation file due to 'random' cmr errors when creating a citation. * added mock testing and retry on 503 * added 503 fixes Co-authored-by: Frank Greguska <Francis.Greguska@jpl.nasa.gov> * fixed issues where token was not proagated to CMR queries (#95) * Misc fixes (#101) * added ".tiff" to default extensions to address #100 * removed 'warning' message on not downloading all data to close #99 * updated help documentation for start/end times to close #79 * added version update, updates to CHANGELOG * added token get,delete, refresh and list operations * Revert "added token get,delete, refresh and list operations" This reverts commit 15aba90. * Update python-app.yml * updated poetry version Version matches build/test versions. * Issues/98 (#107) * added token get,delete, refresh and list operations * Revert "added token get,delete, refresh and list operations" This reverts commit 15aba90. * added EDL (not cmr-token) based get, list,delete, refresh token * updated token regression tests * updates and tests for subscriber moving to EDL. * marked tests as regression test * Update subscriber/podaac_data_downloader.py Co-authored-by: Frank Greguska <89428916+frankinspace@users.noreply.github.com> * Update subscriber/podaac_data_subscriber.py Co-authored-by: Frank Greguska <89428916+frankinspace@users.noreply.github.com> * Update subscriber/podaac_access.py Co-authored-by: Frank Greguska <89428916+frankinspace@users.noreply.github.com> * Update subscriber/podaac_access.py Co-authored-by: Frank Greguska <89428916+frankinspace@users.noreply.github.com> * Update subscriber/podaac_access.py Co-authored-by: Frank Greguska <89428916+frankinspace@users.noreply.github.com> * added exec info to errors, cleaned up some log statements Co-authored-by: Frank Greguska <89428916+frankinspace@users.noreply.github.com> * Issues/109 (#111) * Develop (#103) * Issues/91 (#92) * added citation creation tests and functionality to subscriber and downloader * added verbose option to create_citation_file command, previously hard coded * updated changelog (whoops) and fixed regression test: 1. Issue where the citation file now downloaded affected the counts 2. Issue where the logic for determining if a file modified time was changing or not was picking up the new citation file which _always_ gets rewritten to update the 'last accessed' date. * updated request to include exec_info in warning; fixed issue with params not being a dictionary caused errors * changed a warning to debug for citation file. fixed test issues * Enable debug logging during regression tests and set max parallel workflows to 2 * added output to pytest * fixed test to only look for downlaoded data files not citation file due to 'random' cmr errors when creating a citation. * added mock testing and retry on 503 * added 503 fixes Co-authored-by: Frank Greguska <Francis.Greguska@jpl.nasa.gov> * fixed issues where token was not proagated to CMR queries (#95) * Misc fixes (#101) * added ".tiff" to default extensions to address #100 * removed 'warning' message on not downloading all data to close #99 * updated help documentation for start/end times to close #79 * added version update, updates to CHANGELOG * added token get,delete, refresh and list operations * Revert "added token get,delete, refresh and list operations" This reverts commit 15aba90. * Update python-app.yml Co-authored-by: Frank Greguska <Francis.Greguska@jpl.nasa.gov> * updated poetry version Version matches build/test versions. * Update README.md * Update podaac_data_downloader.py Fixing for issues 109 - adding capability to download by granule-name * Update Downloader.md Fixed the help file * added changelog entries, regressiont ests * added poetry lock cleanup Co-authored-by: Frank Greguska <Francis.Greguska@jpl.nasa.gov> Co-authored-by: Frank Greguska <89428916+frankinspace@users.noreply.github.com> Co-authored-by: sureshshsv <45676320+sureshshsv@users.noreply.github.com> Co-authored-by: sureshshsv <suresh.vannan@jpl.nasa.gov> * added README information and updates (#113) * fixed pymock issues... again Co-authored-by: Frank Greguska <Francis.Greguska@jpl.nasa.gov> Co-authored-by: Frank Greguska <89428916+frankinspace@users.noreply.github.com> Co-authored-by: sureshshsv <45676320+sureshshsv@users.noreply.github.com> Co-authored-by: sureshshsv <suresh.vannan@jpl.nasa.gov>
mike-gangl
added a commit
that referenced
this issue
Jan 30, 2023
* Develop into Main (1.12.0) (#114) * Issues/91 (#92) * added citation creation tests and functionality to subscriber and downloader * added verbose option to create_citation_file command, previously hard coded * updated changelog (whoops) and fixed regression test: 1. Issue where the citation file now downloaded affected the counts 2. Issue where the logic for determining if a file modified time was changing or not was picking up the new citation file which _always_ gets rewritten to update the 'last accessed' date. * updated request to include exec_info in warning; fixed issue with params not being a dictionary caused errors * changed a warning to debug for citation file. fixed test issues * Enable debug logging during regression tests and set max parallel workflows to 2 * added output to pytest * fixed test to only look for downlaoded data files not citation file due to 'random' cmr errors when creating a citation. * added mock testing and retry on 503 * added 503 fixes Co-authored-by: Frank Greguska <Francis.Greguska@jpl.nasa.gov> * fixed issues where token was not proagated to CMR queries (#95) * Misc fixes (#101) * added ".tiff" to default extensions to address #100 * removed 'warning' message on not downloading all data to close #99 * updated help documentation for start/end times to close #79 * added version update, updates to CHANGELOG * added token get,delete, refresh and list operations * Revert "added token get,delete, refresh and list operations" This reverts commit 15aba90. * Update python-app.yml * updated poetry version Version matches build/test versions. * Issues/98 (#107) * added token get,delete, refresh and list operations * Revert "added token get,delete, refresh and list operations" This reverts commit 15aba90. * added EDL (not cmr-token) based get, list,delete, refresh token * updated token regression tests * updates and tests for subscriber moving to EDL. * marked tests as regression test * Update subscriber/podaac_data_downloader.py Co-authored-by: Frank Greguska <89428916+frankinspace@users.noreply.github.com> * Update subscriber/podaac_data_subscriber.py Co-authored-by: Frank Greguska <89428916+frankinspace@users.noreply.github.com> * Update subscriber/podaac_access.py Co-authored-by: Frank Greguska <89428916+frankinspace@users.noreply.github.com> * Update subscriber/podaac_access.py Co-authored-by: Frank Greguska <89428916+frankinspace@users.noreply.github.com> * Update subscriber/podaac_access.py Co-authored-by: Frank Greguska <89428916+frankinspace@users.noreply.github.com> * added exec info to errors, cleaned up some log statements Co-authored-by: Frank Greguska <89428916+frankinspace@users.noreply.github.com> * Issues/109 (#111) * Develop (#103) * Issues/91 (#92) * added citation creation tests and functionality to subscriber and downloader * added verbose option to create_citation_file command, previously hard coded * updated changelog (whoops) and fixed regression test: 1. Issue where the citation file now downloaded affected the counts 2. Issue where the logic for determining if a file modified time was changing or not was picking up the new citation file which _always_ gets rewritten to update the 'last accessed' date. * updated request to include exec_info in warning; fixed issue with params not being a dictionary caused errors * changed a warning to debug for citation file. fixed test issues * Enable debug logging during regression tests and set max parallel workflows to 2 * added output to pytest * fixed test to only look for downlaoded data files not citation file due to 'random' cmr errors when creating a citation. * added mock testing and retry on 503 * added 503 fixes Co-authored-by: Frank Greguska <Francis.Greguska@jpl.nasa.gov> * fixed issues where token was not proagated to CMR queries (#95) * Misc fixes (#101) * added ".tiff" to default extensions to address #100 * removed 'warning' message on not downloading all data to close #99 * updated help documentation for start/end times to close #79 * added version update, updates to CHANGELOG * added token get,delete, refresh and list operations * Revert "added token get,delete, refresh and list operations" This reverts commit 15aba90. * Update python-app.yml Co-authored-by: Frank Greguska <Francis.Greguska@jpl.nasa.gov> * updated poetry version Version matches build/test versions. * Update README.md * Update podaac_data_downloader.py Fixing for issues 109 - adding capability to download by granule-name * Update Downloader.md Fixed the help file * added changelog entries, regressiont ests * added poetry lock cleanup Co-authored-by: Frank Greguska <Francis.Greguska@jpl.nasa.gov> Co-authored-by: Frank Greguska <89428916+frankinspace@users.noreply.github.com> Co-authored-by: sureshshsv <45676320+sureshshsv@users.noreply.github.com> Co-authored-by: sureshshsv <suresh.vannan@jpl.nasa.gov> * added README information and updates (#113) * fixed pymock issues... again Co-authored-by: Frank Greguska <Francis.Greguska@jpl.nasa.gov> Co-authored-by: Frank Greguska <89428916+frankinspace@users.noreply.github.com> Co-authored-by: sureshshsv <45676320+sureshshsv@users.noreply.github.com> Co-authored-by: sureshshsv <suresh.vannan@jpl.nasa.gov> * extend -e option to handle regular expressions formerly, -e could not handle PTM_\d+ extensions without the user explicitly calling all of them. --------- Co-authored-by: mike-gangl <59702631+mike-gangl@users.noreply.github.com> Co-authored-by: Frank Greguska <Francis.Greguska@jpl.nasa.gov> Co-authored-by: Frank Greguska <89428916+frankinspace@users.noreply.github.com> Co-authored-by: sureshshsv <45676320+sureshshsv@users.noreply.github.com> Co-authored-by: sureshshsv <suresh.vannan@jpl.nasa.gov>
mike-gangl
added a commit
that referenced
this issue
Feb 3, 2023
* extend -e option to handle regular expressions (#115) * Develop into Main (1.12.0) (#114) * Issues/91 (#92) * added citation creation tests and functionality to subscriber and downloader * added verbose option to create_citation_file command, previously hard coded * updated changelog (whoops) and fixed regression test: 1. Issue where the citation file now downloaded affected the counts 2. Issue where the logic for determining if a file modified time was changing or not was picking up the new citation file which _always_ gets rewritten to update the 'last accessed' date. * updated request to include exec_info in warning; fixed issue with params not being a dictionary caused errors * changed a warning to debug for citation file. fixed test issues * Enable debug logging during regression tests and set max parallel workflows to 2 * added output to pytest * fixed test to only look for downlaoded data files not citation file due to 'random' cmr errors when creating a citation. * added mock testing and retry on 503 * added 503 fixes Co-authored-by: Frank Greguska <Francis.Greguska@jpl.nasa.gov> * fixed issues where token was not proagated to CMR queries (#95) * Misc fixes (#101) * added ".tiff" to default extensions to address #100 * removed 'warning' message on not downloading all data to close #99 * updated help documentation for start/end times to close #79 * added version update, updates to CHANGELOG * added token get,delete, refresh and list operations * Revert "added token get,delete, refresh and list operations" This reverts commit 15aba90. * Update python-app.yml * updated poetry version Version matches build/test versions. * Issues/98 (#107) * added token get,delete, refresh and list operations * Revert "added token get,delete, refresh and list operations" This reverts commit 15aba90. * added EDL (not cmr-token) based get, list,delete, refresh token * updated token regression tests * updates and tests for subscriber moving to EDL. * marked tests as regression test * Update subscriber/podaac_data_downloader.py Co-authored-by: Frank Greguska <89428916+frankinspace@users.noreply.github.com> * Update subscriber/podaac_data_subscriber.py Co-authored-by: Frank Greguska <89428916+frankinspace@users.noreply.github.com> * Update subscriber/podaac_access.py Co-authored-by: Frank Greguska <89428916+frankinspace@users.noreply.github.com> * Update subscriber/podaac_access.py Co-authored-by: Frank Greguska <89428916+frankinspace@users.noreply.github.com> * Update subscriber/podaac_access.py Co-authored-by: Frank Greguska <89428916+frankinspace@users.noreply.github.com> * added exec info to errors, cleaned up some log statements Co-authored-by: Frank Greguska <89428916+frankinspace@users.noreply.github.com> * Issues/109 (#111) * Develop (#103) * Issues/91 (#92) * added citation creation tests and functionality to subscriber and downloader * added verbose option to create_citation_file command, previously hard coded * updated changelog (whoops) and fixed regression test: 1. Issue where the citation file now downloaded affected the counts 2. Issue where the logic for determining if a file modified time was changing or not was picking up the new citation file which _always_ gets rewritten to update the 'last accessed' date. * updated request to include exec_info in warning; fixed issue with params not being a dictionary caused errors * changed a warning to debug for citation file. fixed test issues * Enable debug logging during regression tests and set max parallel workflows to 2 * added output to pytest * fixed test to only look for downlaoded data files not citation file due to 'random' cmr errors when creating a citation. * added mock testing and retry on 503 * added 503 fixes Co-authored-by: Frank Greguska <Francis.Greguska@jpl.nasa.gov> * fixed issues where token was not proagated to CMR queries (#95) * Misc fixes (#101) * added ".tiff" to default extensions to address #100 * removed 'warning' message on not downloading all data to close #99 * updated help documentation for start/end times to close #79 * added version update, updates to CHANGELOG * added token get,delete, refresh and list operations * Revert "added token get,delete, refresh and list operations" This reverts commit 15aba90. * Update python-app.yml Co-authored-by: Frank Greguska <Francis.Greguska@jpl.nasa.gov> * updated poetry version Version matches build/test versions. * Update README.md * Update podaac_data_downloader.py Fixing for issues 109 - adding capability to download by granule-name * Update Downloader.md Fixed the help file * added changelog entries, regressiont ests * added poetry lock cleanup Co-authored-by: Frank Greguska <Francis.Greguska@jpl.nasa.gov> Co-authored-by: Frank Greguska <89428916+frankinspace@users.noreply.github.com> Co-authored-by: sureshshsv <45676320+sureshshsv@users.noreply.github.com> Co-authored-by: sureshshsv <suresh.vannan@jpl.nasa.gov> * added README information and updates (#113) * fixed pymock issues... again Co-authored-by: Frank Greguska <Francis.Greguska@jpl.nasa.gov> Co-authored-by: Frank Greguska <89428916+frankinspace@users.noreply.github.com> Co-authored-by: sureshshsv <45676320+sureshshsv@users.noreply.github.com> Co-authored-by: sureshshsv <suresh.vannan@jpl.nasa.gov> * extend -e option to handle regular expressions formerly, -e could not handle PTM_\d+ extensions without the user explicitly calling all of them. --------- Co-authored-by: mike-gangl <59702631+mike-gangl@users.noreply.github.com> Co-authored-by: Frank Greguska <Francis.Greguska@jpl.nasa.gov> Co-authored-by: Frank Greguska <89428916+frankinspace@users.noreply.github.com> Co-authored-by: sureshshsv <45676320+sureshshsv@users.noreply.github.com> Co-authored-by: sureshshsv <suresh.vannan@jpl.nasa.gov> * added dcoumentation and tests for regex * converted defaults to regexes, added gtiff test --------- Co-authored-by: Peter Mao <peter.mao@gmail.com> Co-authored-by: Frank Greguska <Francis.Greguska@jpl.nasa.gov> Co-authored-by: Frank Greguska <89428916+frankinspace@users.noreply.github.com> Co-authored-by: sureshshsv <45676320+sureshshsv@users.noreply.github.com> Co-authored-by: sureshshsv <suresh.vannan@jpl.nasa.gov>
mike-gangl
added a commit
that referenced
this issue
Apr 26, 2023
* Issues/91 (#92) * added citation creation tests and functionality to subscriber and downloader * added verbose option to create_citation_file command, previously hard coded * updated changelog (whoops) and fixed regression test: 1. Issue where the citation file now downloaded affected the counts 2. Issue where the logic for determining if a file modified time was changing or not was picking up the new citation file which _always_ gets rewritten to update the 'last accessed' date. * updated request to include exec_info in warning; fixed issue with params not being a dictionary caused errors * changed a warning to debug for citation file. fixed test issues * Enable debug logging during regression tests and set max parallel workflows to 2 * added output to pytest * fixed test to only look for downlaoded data files not citation file due to 'random' cmr errors when creating a citation. * added mock testing and retry on 503 * added 503 fixes Co-authored-by: Frank Greguska <Francis.Greguska@jpl.nasa.gov> * fixed issues where token was not proagated to CMR queries (#95) * Misc fixes (#101) * added ".tiff" to default extensions to address #100 * removed 'warning' message on not downloading all data to close #99 * updated help documentation for start/end times to close #79 * added version update, updates to CHANGELOG * added token get,delete, refresh and list operations * Revert "added token get,delete, refresh and list operations" This reverts commit 15aba90. * Update python-app.yml * updated poetry version Version matches build/test versions. * Issues/98 (#107) * added token get,delete, refresh and list operations * Revert "added token get,delete, refresh and list operations" This reverts commit 15aba90. * added EDL (not cmr-token) based get, list,delete, refresh token * updated token regression tests * updates and tests for subscriber moving to EDL. * marked tests as regression test * Update subscriber/podaac_data_downloader.py Co-authored-by: Frank Greguska <89428916+frankinspace@users.noreply.github.com> * Update subscriber/podaac_data_subscriber.py Co-authored-by: Frank Greguska <89428916+frankinspace@users.noreply.github.com> * Update subscriber/podaac_access.py Co-authored-by: Frank Greguska <89428916+frankinspace@users.noreply.github.com> * Update subscriber/podaac_access.py Co-authored-by: Frank Greguska <89428916+frankinspace@users.noreply.github.com> * Update subscriber/podaac_access.py Co-authored-by: Frank Greguska <89428916+frankinspace@users.noreply.github.com> * added exec info to errors, cleaned up some log statements Co-authored-by: Frank Greguska <89428916+frankinspace@users.noreply.github.com> * Issues/109 (#111) * Develop (#103) * Issues/91 (#92) * added citation creation tests and functionality to subscriber and downloader * added verbose option to create_citation_file command, previously hard coded * updated changelog (whoops) and fixed regression test: 1. Issue where the citation file now downloaded affected the counts 2. Issue where the logic for determining if a file modified time was changing or not was picking up the new citation file which _always_ gets rewritten to update the 'last accessed' date. * updated request to include exec_info in warning; fixed issue with params not being a dictionary caused errors * changed a warning to debug for citation file. fixed test issues * Enable debug logging during regression tests and set max parallel workflows to 2 * added output to pytest * fixed test to only look for downlaoded data files not citation file due to 'random' cmr errors when creating a citation. * added mock testing and retry on 503 * added 503 fixes Co-authored-by: Frank Greguska <Francis.Greguska@jpl.nasa.gov> * fixed issues where token was not proagated to CMR queries (#95) * Misc fixes (#101) * added ".tiff" to default extensions to address #100 * removed 'warning' message on not downloading all data to close #99 * updated help documentation for start/end times to close #79 * added version update, updates to CHANGELOG * added token get,delete, refresh and list operations * Revert "added token get,delete, refresh and list operations" This reverts commit 15aba90. * Update python-app.yml Co-authored-by: Frank Greguska <Francis.Greguska@jpl.nasa.gov> * updated poetry version Version matches build/test versions. * Update README.md * Update podaac_data_downloader.py Fixing for issues 109 - adding capability to download by granule-name * Update Downloader.md Fixed the help file * added changelog entries, regressiont ests * added poetry lock cleanup Co-authored-by: Frank Greguska <Francis.Greguska@jpl.nasa.gov> Co-authored-by: Frank Greguska <89428916+frankinspace@users.noreply.github.com> Co-authored-by: sureshshsv <45676320+sureshshsv@users.noreply.github.com> Co-authored-by: sureshshsv <suresh.vannan@jpl.nasa.gov> * added README information and updates (#113) * fixed pymock issues... again * Extension regex (#121) * extend -e option to handle regular expressions (#115) * Develop into Main (1.12.0) (#114) * Issues/91 (#92) * added citation creation tests and functionality to subscriber and downloader * added verbose option to create_citation_file command, previously hard coded * updated changelog (whoops) and fixed regression test: 1. Issue where the citation file now downloaded affected the counts 2. Issue where the logic for determining if a file modified time was changing or not was picking up the new citation file which _always_ gets rewritten to update the 'last accessed' date. * updated request to include exec_info in warning; fixed issue with params not being a dictionary caused errors * changed a warning to debug for citation file. fixed test issues * Enable debug logging during regression tests and set max parallel workflows to 2 * added output to pytest * fixed test to only look for downlaoded data files not citation file due to 'random' cmr errors when creating a citation. * added mock testing and retry on 503 * added 503 fixes Co-authored-by: Frank Greguska <Francis.Greguska@jpl.nasa.gov> * fixed issues where token was not proagated to CMR queries (#95) * Misc fixes (#101) * added ".tiff" to default extensions to address #100 * removed 'warning' message on not downloading all data to close #99 * updated help documentation for start/end times to close #79 * added version update, updates to CHANGELOG * added token get,delete, refresh and list operations * Revert "added token get,delete, refresh and list operations" This reverts commit 15aba90. * Update python-app.yml * updated poetry version Version matches build/test versions. * Issues/98 (#107) * added token get,delete, refresh and list operations * Revert "added token get,delete, refresh and list operations" This reverts commit 15aba90. * added EDL (not cmr-token) based get, list,delete, refresh token * updated token regression tests * updates and tests for subscriber moving to EDL. * marked tests as regression test * Update subscriber/podaac_data_downloader.py Co-authored-by: Frank Greguska <89428916+frankinspace@users.noreply.github.com> * Update subscriber/podaac_data_subscriber.py Co-authored-by: Frank Greguska <89428916+frankinspace@users.noreply.github.com> * Update subscriber/podaac_access.py Co-authored-by: Frank Greguska <89428916+frankinspace@users.noreply.github.com> * Update subscriber/podaac_access.py Co-authored-by: Frank Greguska <89428916+frankinspace@users.noreply.github.com> * Update subscriber/podaac_access.py Co-authored-by: Frank Greguska <89428916+frankinspace@users.noreply.github.com> * added exec info to errors, cleaned up some log statements Co-authored-by: Frank Greguska <89428916+frankinspace@users.noreply.github.com> * Issues/109 (#111) * Develop (#103) * Issues/91 (#92) * added citation creation tests and functionality to subscriber and downloader * added verbose option to create_citation_file command, previously hard coded * updated changelog (whoops) and fixed regression test: 1. Issue where the citation file now downloaded affected the counts 2. Issue where the logic for determining if a file modified time was changing or not was picking up the new citation file which _always_ gets rewritten to update the 'last accessed' date. * updated request to include exec_info in warning; fixed issue with params not being a dictionary caused errors * changed a warning to debug for citation file. fixed test issues * Enable debug logging during regression tests and set max parallel workflows to 2 * added output to pytest * fixed test to only look for downlaoded data files not citation file due to 'random' cmr errors when creating a citation. * added mock testing and retry on 503 * added 503 fixes Co-authored-by: Frank Greguska <Francis.Greguska@jpl.nasa.gov> * fixed issues where token was not proagated to CMR queries (#95) * Misc fixes (#101) * added ".tiff" to default extensions to address #100 * removed 'warning' message on not downloading all data to close #99 * updated help documentation for start/end times to close #79 * added version update, updates to CHANGELOG * added token get,delete, refresh and list operations * Revert "added token get,delete, refresh and list operations" This reverts commit 15aba90. * Update python-app.yml Co-authored-by: Frank Greguska <Francis.Greguska@jpl.nasa.gov> * updated poetry version Version matches build/test versions. * Update README.md * Update podaac_data_downloader.py Fixing for issues 109 - adding capability to download by granule-name * Update Downloader.md Fixed the help file * added changelog entries, regressiont ests * added poetry lock cleanup Co-authored-by: Frank Greguska <Francis.Greguska@jpl.nasa.gov> Co-authored-by: Frank Greguska <89428916+frankinspace@users.noreply.github.com> Co-authored-by: sureshshsv <45676320+sureshshsv@users.noreply.github.com> Co-authored-by: sureshshsv <suresh.vannan@jpl.nasa.gov> * added README information and updates (#113) * fixed pymock issues... again Co-authored-by: Frank Greguska <Francis.Greguska@jpl.nasa.gov> Co-authored-by: Frank Greguska <89428916+frankinspace@users.noreply.github.com> Co-authored-by: sureshshsv <45676320+sureshshsv@users.noreply.github.com> Co-authored-by: sureshshsv <suresh.vannan@jpl.nasa.gov> * extend -e option to handle regular expressions formerly, -e could not handle PTM_\d+ extensions without the user explicitly calling all of them. --------- Co-authored-by: mike-gangl <59702631+mike-gangl@users.noreply.github.com> Co-authored-by: Frank Greguska <Francis.Greguska@jpl.nasa.gov> Co-authored-by: Frank Greguska <89428916+frankinspace@users.noreply.github.com> Co-authored-by: sureshshsv <45676320+sureshshsv@users.noreply.github.com> Co-authored-by: sureshshsv <suresh.vannan@jpl.nasa.gov> * added dcoumentation and tests for regex * converted defaults to regexes, added gtiff test --------- Co-authored-by: Peter Mao <peter.mao@gmail.com> Co-authored-by: Frank Greguska <Francis.Greguska@jpl.nasa.gov> Co-authored-by: Frank Greguska <89428916+frankinspace@users.noreply.github.com> Co-authored-by: sureshshsv <45676320+sureshshsv@users.noreply.github.com> Co-authored-by: sureshshsv <suresh.vannan@jpl.nasa.gov> * closes 118. retries was never hit because range is not end inclusive. (#119) * closes 118. retries was never hit ebcause range is not end inclusive. * updated test to catch now-thrown exception * added --dry-run option, docs, and test cases (#124) * added --dry-run option, docs, and test cases * Update subscriber/podaac_data_downloader.py Added more elegant way of download limit application Co-authored-by: Stepheny Perez <skorper@users.noreply.github.com> --------- Co-authored-by: Stepheny Perez <skorper@users.noreply.github.com> * Issues/70 (#117) * added code for updating version * added chagnelog * moved version check into __main__ instead of on import of the module * added sorting of releases from github to find latest release. * added authenticated (option) access to github API to rpevent rate limiting * separate out auth/token regression tests * Issues/127 (#128) * added token sensitivity filter to remove tokens from CMR queries * added changelog updates * updated some lingering merge issues (huh?) * updated regression test * updated ubuntu versions * removed 18.04 ubuntu from workflows/actions * version and documentation updates (#130) --------- Co-authored-by: Frank Greguska <Francis.Greguska@jpl.nasa.gov> Co-authored-by: Frank Greguska <89428916+frankinspace@users.noreply.github.com> Co-authored-by: sureshshsv <45676320+sureshshsv@users.noreply.github.com> Co-authored-by: sureshshsv <suresh.vannan@jpl.nasa.gov> Co-authored-by: Peter Mao <peter.mao@gmail.com> Co-authored-by: Stepheny Perez <skorper@users.noreply.github.com>
mike-gangl
added a commit
that referenced
this issue
Apr 28, 2023
* Issues/91 (#92) * added citation creation tests and functionality to subscriber and downloader * added verbose option to create_citation_file command, previously hard coded * updated changelog (whoops) and fixed regression test: 1. Issue where the citation file now downloaded affected the counts 2. Issue where the logic for determining if a file modified time was changing or not was picking up the new citation file which _always_ gets rewritten to update the 'last accessed' date. * updated request to include exec_info in warning; fixed issue with params not being a dictionary caused errors * changed a warning to debug for citation file. fixed test issues * Enable debug logging during regression tests and set max parallel workflows to 2 * added output to pytest * fixed test to only look for downlaoded data files not citation file due to 'random' cmr errors when creating a citation. * added mock testing and retry on 503 * added 503 fixes Co-authored-by: Frank Greguska <Francis.Greguska@jpl.nasa.gov> * fixed issues where token was not proagated to CMR queries (#95) * Misc fixes (#101) * added ".tiff" to default extensions to address #100 * removed 'warning' message on not downloading all data to close #99 * updated help documentation for start/end times to close #79 * added version update, updates to CHANGELOG * added token get,delete, refresh and list operations * Revert "added token get,delete, refresh and list operations" This reverts commit 15aba90. * Update python-app.yml * updated poetry version Version matches build/test versions. * Issues/98 (#107) * added token get,delete, refresh and list operations * Revert "added token get,delete, refresh and list operations" This reverts commit 15aba90. * added EDL (not cmr-token) based get, list,delete, refresh token * updated token regression tests * updates and tests for subscriber moving to EDL. * marked tests as regression test * Update subscriber/podaac_data_downloader.py Co-authored-by: Frank Greguska <89428916+frankinspace@users.noreply.github.com> * Update subscriber/podaac_data_subscriber.py Co-authored-by: Frank Greguska <89428916+frankinspace@users.noreply.github.com> * Update subscriber/podaac_access.py Co-authored-by: Frank Greguska <89428916+frankinspace@users.noreply.github.com> * Update subscriber/podaac_access.py Co-authored-by: Frank Greguska <89428916+frankinspace@users.noreply.github.com> * Update subscriber/podaac_access.py Co-authored-by: Frank Greguska <89428916+frankinspace@users.noreply.github.com> * added exec info to errors, cleaned up some log statements Co-authored-by: Frank Greguska <89428916+frankinspace@users.noreply.github.com> * Issues/109 (#111) * Develop (#103) * Issues/91 (#92) * added citation creation tests and functionality to subscriber and downloader * added verbose option to create_citation_file command, previously hard coded * updated changelog (whoops) and fixed regression test: 1. Issue where the citation file now downloaded affected the counts 2. Issue where the logic for determining if a file modified time was changing or not was picking up the new citation file which _always_ gets rewritten to update the 'last accessed' date. * updated request to include exec_info in warning; fixed issue with params not being a dictionary caused errors * changed a warning to debug for citation file. fixed test issues * Enable debug logging during regression tests and set max parallel workflows to 2 * added output to pytest * fixed test to only look for downlaoded data files not citation file due to 'random' cmr errors when creating a citation. * added mock testing and retry on 503 * added 503 fixes Co-authored-by: Frank Greguska <Francis.Greguska@jpl.nasa.gov> * fixed issues where token was not proagated to CMR queries (#95) * Misc fixes (#101) * added ".tiff" to default extensions to address #100 * removed 'warning' message on not downloading all data to close #99 * updated help documentation for start/end times to close #79 * added version update, updates to CHANGELOG * added token get,delete, refresh and list operations * Revert "added token get,delete, refresh and list operations" This reverts commit 15aba90. * Update python-app.yml Co-authored-by: Frank Greguska <Francis.Greguska@jpl.nasa.gov> * updated poetry version Version matches build/test versions. * Update README.md * Update podaac_data_downloader.py Fixing for issues 109 - adding capability to download by granule-name * Update Downloader.md Fixed the help file * added changelog entries, regressiont ests * added poetry lock cleanup Co-authored-by: Frank Greguska <Francis.Greguska@jpl.nasa.gov> Co-authored-by: Frank Greguska <89428916+frankinspace@users.noreply.github.com> Co-authored-by: sureshshsv <45676320+sureshshsv@users.noreply.github.com> Co-authored-by: sureshshsv <suresh.vannan@jpl.nasa.gov> * added README information and updates (#113) * fixed pymock issues... again * Extension regex (#121) * extend -e option to handle regular expressions (#115) * Develop into Main (1.12.0) (#114) * Issues/91 (#92) * added citation creation tests and functionality to subscriber and downloader * added verbose option to create_citation_file command, previously hard coded * updated changelog (whoops) and fixed regression test: 1. Issue where the citation file now downloaded affected the counts 2. Issue where the logic for determining if a file modified time was changing or not was picking up the new citation file which _always_ gets rewritten to update the 'last accessed' date. * updated request to include exec_info in warning; fixed issue with params not being a dictionary caused errors * changed a warning to debug for citation file. fixed test issues * Enable debug logging during regression tests and set max parallel workflows to 2 * added output to pytest * fixed test to only look for downlaoded data files not citation file due to 'random' cmr errors when creating a citation. * added mock testing and retry on 503 * added 503 fixes Co-authored-by: Frank Greguska <Francis.Greguska@jpl.nasa.gov> * fixed issues where token was not proagated to CMR queries (#95) * Misc fixes (#101) * added ".tiff" to default extensions to address #100 * removed 'warning' message on not downloading all data to close #99 * updated help documentation for start/end times to close #79 * added version update, updates to CHANGELOG * added token get,delete, refresh and list operations * Revert "added token get,delete, refresh and list operations" This reverts commit 15aba90. * Update python-app.yml * updated poetry version Version matches build/test versions. * Issues/98 (#107) * added token get,delete, refresh and list operations * Revert "added token get,delete, refresh and list operations" This reverts commit 15aba90. * added EDL (not cmr-token) based get, list,delete, refresh token * updated token regression tests * updates and tests for subscriber moving to EDL. * marked tests as regression test * Update subscriber/podaac_data_downloader.py Co-authored-by: Frank Greguska <89428916+frankinspace@users.noreply.github.com> * Update subscriber/podaac_data_subscriber.py Co-authored-by: Frank Greguska <89428916+frankinspace@users.noreply.github.com> * Update subscriber/podaac_access.py Co-authored-by: Frank Greguska <89428916+frankinspace@users.noreply.github.com> * Update subscriber/podaac_access.py Co-authored-by: Frank Greguska <89428916+frankinspace@users.noreply.github.com> * Update subscriber/podaac_access.py Co-authored-by: Frank Greguska <89428916+frankinspace@users.noreply.github.com> * added exec info to errors, cleaned up some log statements Co-authored-by: Frank Greguska <89428916+frankinspace@users.noreply.github.com> * Issues/109 (#111) * Develop (#103) * Issues/91 (#92) * added citation creation tests and functionality to subscriber and downloader * added verbose option to create_citation_file command, previously hard coded * updated changelog (whoops) and fixed regression test: 1. Issue where the citation file now downloaded affected the counts 2. Issue where the logic for determining if a file modified time was changing or not was picking up the new citation file which _always_ gets rewritten to update the 'last accessed' date. * updated request to include exec_info in warning; fixed issue with params not being a dictionary caused errors * changed a warning to debug for citation file. fixed test issues * Enable debug logging during regression tests and set max parallel workflows to 2 * added output to pytest * fixed test to only look for downlaoded data files not citation file due to 'random' cmr errors when creating a citation. * added mock testing and retry on 503 * added 503 fixes Co-authored-by: Frank Greguska <Francis.Greguska@jpl.nasa.gov> * fixed issues where token was not proagated to CMR queries (#95) * Misc fixes (#101) * added ".tiff" to default extensions to address #100 * removed 'warning' message on not downloading all data to close #99 * updated help documentation for start/end times to close #79 * added version update, updates to CHANGELOG * added token get,delete, refresh and list operations * Revert "added token get,delete, refresh and list operations" This reverts commit 15aba90. * Update python-app.yml Co-authored-by: Frank Greguska <Francis.Greguska@jpl.nasa.gov> * updated poetry version Version matches build/test versions. * Update README.md * Update podaac_data_downloader.py Fixing for issues 109 - adding capability to download by granule-name * Update Downloader.md Fixed the help file * added changelog entries, regressiont ests * added poetry lock cleanup Co-authored-by: Frank Greguska <Francis.Greguska@jpl.nasa.gov> Co-authored-by: Frank Greguska <89428916+frankinspace@users.noreply.github.com> Co-authored-by: sureshshsv <45676320+sureshshsv@users.noreply.github.com> Co-authored-by: sureshshsv <suresh.vannan@jpl.nasa.gov> * added README information and updates (#113) * fixed pymock issues... again Co-authored-by: Frank Greguska <Francis.Greguska@jpl.nasa.gov> Co-authored-by: Frank Greguska <89428916+frankinspace@users.noreply.github.com> Co-authored-by: sureshshsv <45676320+sureshshsv@users.noreply.github.com> Co-authored-by: sureshshsv <suresh.vannan@jpl.nasa.gov> * extend -e option to handle regular expressions formerly, -e could not handle PTM_\d+ extensions without the user explicitly calling all of them. --------- Co-authored-by: mike-gangl <59702631+mike-gangl@users.noreply.github.com> Co-authored-by: Frank Greguska <Francis.Greguska@jpl.nasa.gov> Co-authored-by: Frank Greguska <89428916+frankinspace@users.noreply.github.com> Co-authored-by: sureshshsv <45676320+sureshshsv@users.noreply.github.com> Co-authored-by: sureshshsv <suresh.vannan@jpl.nasa.gov> * added dcoumentation and tests for regex * converted defaults to regexes, added gtiff test --------- Co-authored-by: Peter Mao <peter.mao@gmail.com> Co-authored-by: Frank Greguska <Francis.Greguska@jpl.nasa.gov> Co-authored-by: Frank Greguska <89428916+frankinspace@users.noreply.github.com> Co-authored-by: sureshshsv <45676320+sureshshsv@users.noreply.github.com> Co-authored-by: sureshshsv <suresh.vannan@jpl.nasa.gov> * closes 118. retries was never hit because range is not end inclusive. (#119) * closes 118. retries was never hit ebcause range is not end inclusive. * updated test to catch now-thrown exception * added --dry-run option, docs, and test cases (#124) * added --dry-run option, docs, and test cases * Update subscriber/podaac_data_downloader.py Added more elegant way of download limit application Co-authored-by: Stepheny Perez <skorper@users.noreply.github.com> --------- Co-authored-by: Stepheny Perez <skorper@users.noreply.github.com> * Issues/70 (#117) * added code for updating version * added chagnelog * moved version check into __main__ instead of on import of the module * added sorting of releases from github to find latest release. * added authenticated (option) access to github API to rpevent rate limiting * separate out auth/token regression tests * Issues/127 (#128) * added token sensitivity filter to remove tokens from CMR queries * added changelog updates * updated some lingering merge issues (huh?) * updated regression test * updated ubuntu versions * removed 18.04 ubuntu from workflows/actions * version and documentation updates (#130) * 1.13.1 changelog and dependecny updates * fixed formatting from unsaved merges --------- Co-authored-by: Frank Greguska <Francis.Greguska@jpl.nasa.gov> Co-authored-by: Frank Greguska <89428916+frankinspace@users.noreply.github.com> Co-authored-by: sureshshsv <45676320+sureshshsv@users.noreply.github.com> Co-authored-by: sureshshsv <suresh.vannan@jpl.nasa.gov> Co-authored-by: Peter Mao <peter.mao@gmail.com> Co-authored-by: Stepheny Perez <skorper@users.noreply.github.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Example log
A warning is printed to the log when downloading many granules that says only the first 2000 will be downloaded. But all granules get downloaded.
Limits were removed in #65 so this warning should likely be removed.
The text was updated successfully, but these errors were encountered: