Add MIRS reader #1486
Closed
Add MIRS reader #1486
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Includes hack for hanging test that is yet to be solved
…e formats for FCI L2 PF products
…e formats for FCI L2 PF products
Add support for s3 buckets in OLCI and ABI l1 readers
Removes alteration on ABI BT enhancements
1.) Multiple changes to support use of dask.array.map_blocks rather than memmap for limb correction 2.) BUG FIX to selection of data based on DataID 3.) Added test code Merge branch 'mirs_xarray' of github.com:joleenf/satpy into mirs_reader
1.) Multiple changes to support use of dask.array.map_blocks rather than memmap for limb correction 2.) BUG FIX to selection of data based on DataID 3.) Added test code Merge branch 'mirs_xarray' of github.com:joleenf/satpy into mirs_reader
etc/readers/mirs_l2_nc.yaml
1.) use [longitude, latitude] for consistency with mirs reader code
2.) add long_name and use name to be same as file_key because two
DataIDs were getting set and only one was working. (what now
is long_name was not working as name)
readers/mirs_l2_nc.py
1.) Change variables name of "Latitude" and "Longitude"
to "latitude" and "longitude" so that coordinate variables
are longitude and latitude, because the mixed case version
was not getting consitently recognized as coordinates...and
it could be the way I was defining the coordinate variables
at the time.
2.) Make sure coordinate attributes are added when the available
datasets are built. This helps later when the SwathDefinition
automatically is created.
3.) Add a CRS in data attributes for xrImage. Use WGS84, CRS(4326),
consistent with the SwathDefinition default.
tests/reader_tests/test_mirs_l2_nc.py
Don't make test arrays so complicated.
etc/readers/mirs_l2_nc.yaml
1.) use [longitude, latitude] for consistency with mirs reader code
2.) add long_name and use name to be same as file_key because two
DataIDs were getting set and only one was working. (what now
is long_name was not working as name)
readers/mirs_l2_nc.py
1.) Change variables name of "Latitude" and "Longitude"
to "latitude" and "longitude" so that coordinate variables
are longitude and latitude, because the mixed case version
was not getting consitently recognized as coordinates...and
it could be the way I was defining the coordinate variables
at the time.
2.) Make sure coordinate attributes are added when the available
datasets are built. This helps later when the SwathDefinition
automatically is created.
3.) Add a CRS in data attributes for xrImage. Use WGS84, CRS(4326),
consistent with the SwathDefinition default.
tests/reader_tests/test_mirs_l2_nc.py
Don't make test arrays so complicated.
… BT.
1.) Single dimension coordinates like Freq cannot be mapped in raw form.
2.) A little work needs to be done to make 3D vars other than BT displayable.
It is possible, but also adds another layer to the code. Since they do
not need to be supported don't add for now.
3.) Make sure code that skips limb correction for the short term does not
introduce a bug when uncommented or limb correction is true. (Therefore,
the changes to where bt_data is reduced to 2D and the do_not_apply
boolean.
… BT.
1.) Single dimension coordinates like Freq cannot be mapped in raw form.
2.) A little work needs to be done to make 3D vars other than BT displayable.
It is possible, but also adds another layer to the code. Since they do
not need to be supported don't add for now.
3.) Make sure code that skips limb correction for the short term does not
introduce a bug when uncommented or limb correction is true. (Therefore,
the changes to where bt_data is reduced to 2D and the do_not_apply
boolean.
This was causing issues because one of the coordinates was being found twice for DataIds and that was messy. Just take it out, the coordinates should be found in the reader code.
This was causing issues because one of the coordinates was being found twice for DataIds and that was messy. Just take it out, the coordinates should be found in the reader code.
Previous to this, the date and time were split causing find_files_and_readers to interpret the datetime of the file as datetime(1900,1,1,16,1) and that is not correct. This would cause the reader to miss files when start_time/end_time combinations were used for metop files. This commit parses the date+"strings in between"+ time as one so that the start_time can be interpreted properly as something like dateime(2017,2,6,16,1) <== that being the example from the test file... IMG_SX.M2.D17037.S1601.E1607.B0000001.WE.HR.ORB.nc
Previous to this, the date and time were split causing find_files_and_readers to interpret the datetime of the file as datetime(1900,1,1,16,1) and that is not correct. This would cause the reader to miss files when start_time/end_time combinations were used for metop files. This commit parses the date+"strings in between"+ time as one so that the start_time can be interpreted properly as something like dateime(2017,2,6,16,1) <== that being the example from the test file... IMG_SX.M2.D17037.S1601.E1607.B0000001.WE.HR.ORB.nc
djhoese
requested changes
Jan 20, 2021
satpy/readers/mirs_l2_nc.py
Outdated
| from pyproj import CRS | ||
| from satpy.readers.file_handlers import BaseFileHandler | ||
|
|
||
| LOAD_CHUNK_SIZE = int(os.getenv('PYTROLL_LOAD_CHUNK_SIZE', -1)) |
There was a problem hiding this comment.
This should be taken from satpy import CHUNK_SIZE so it can be updated more easily in the future. The environment variable is PYTROLL_CHUNK_SIZE that controls that value.
satpy/readers/mirs_l2_nc.py
Outdated
|
|
||
| return new_ds | ||
|
|
||
| @staticmethod |
There was a problem hiding this comment.
There are some cases (multi-process scheduler) where dask doesn't always work well with functions that aren't globally defined. I'm not sure if staticmethods have this problem, but what are your thoughts on moving this staticmethod (and any other delayed/map_blocks'd methods) to be functions in the global scope?
1.) CHUNK_SIZE is used to make future updates easier 2.) Move dask operations out of staticmethod to a global method Reason: There are cases (multi-process scheduler) when dask doesn't work well with functions that aren't globally defined. 3.) BUG FIX: noaa-20 should be atms sensor, so that limb correction is applied as promised.
1.) CHUNK_SIZE is used to make future updates easier 2.) Move dask operations out of staticmethod to a global method Reason: There are cases (multi-process scheduler) when dask doesn't work well with functions that aren't globally defined. 3.) BUG FIX: noaa-20 should be atms sensor, so that limb correction is applied as promised.
like the you say potato platitude.
like the you say potato platitude.
This reverts commit 841c0e4.
filter-branch Merge branch 'mirs_reader' of github.com:joleenf/satpy into mirs_reader
|
Hi @djhoese, I have moved the limb_correction code to a global function. CHUNK_SIZE is being used. I believe I have removed all of the correction files that I have inadvertently committed. The correction is currently not active due to the fact that we need to figure out where those correction files will be stored. |
|
@djhoese I probably should have left this as a draft considering the question of coeffcient files. |
|
Hm it seems you have confused github (6,859 commits). Yay git! It looks like the commands you ran to remove the files from your commits must have gone through and edited almost all Satpy commits (master has ~8k commits). If you can't figure it out, it may be worth copying your changed files to a new branch and making a new PR. Sorry. |
|
Additionally it looks like one of the test environments failed. |
|
Oh. No. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
flake8 satpy