Refactor Sentinel-1 SAR-C reader #2727
Conversation
|
pre-commit.ci autofix |
for more information, see https://pre-commit.ci
| def create_storage_items(self, files, **kwargs): | ||
| """Create the storage items.""" |
There was a problem hiding this comment.
I'm not entirely sure I understand what happens here, but it looks more like file handler creation rather than collecting the stored data. Maybe naming and docstring could be updated/expanded a bit? What are the "items"?
There was a problem hiding this comment.
The idea here was to have a function name that would be more generic than filehandlers. Hence the renaming to storage items.
In the case of this reader, with still have a loose usage of filehandlers, but in principle a dev implementing a new reader would use this function just to initialise storage "items", or "units" or "objects" from the filenames.
| @property | ||
| def start_time(self): | ||
| """Get the start time.""" | ||
| return self.storage_items.values()[0].filename_info["start_time"].replace(tzinfo=tz.utc) | ||
|
|
||
| @property | ||
| def end_time(self): | ||
| """Get the end time.""" | ||
| return self.storage_items.values()[0].filename_info["end_time"].replace(tzinfo=tz.utc) |
There was a problem hiding this comment.
These look a bit costly (maybe, see my comment on storage item collection), should these be cached properties?
There was a problem hiding this comment.
I'm not sure about this, as in theory the reader object is not immutable... a cached property would make the assumption that the files/storage items never changes.
I know that in practice, this never happens, but this may still be confusing and bug-prone
There was a problem hiding this comment.
Is this different to other (traditional) readers and one can append more data to it? Or what would be the mechanism that changed the times after Scene creation?
There was a problem hiding this comment.
no, it's the same compared from the reader level (not the filehandler). In theory, nothing prevents a user to call create_filehandlers (now create_storage_items) multiple times on the same reader...
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #2727 +/- ##
========================================
Coverage 95.93% 95.94%
========================================
Files 375 375
Lines 53237 53456 +219
========================================
+ Hits 51075 51290 +215
- Misses 2162 2166 +4
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. |
Pull Request Test Coverage Report for Build 8879886090Warning: This coverage report may be inaccurate.This pull request's base commit is no longer the HEAD commit of its target branch. This means it includes changes from outside the original pull request, including, potentially, unrelated coverage changes.
Details
💛 - Coveralls |
|
Merging is fine from my point of view. |
There was a problem hiding this comment.
I have a couple requests, but otherwise I just wanted to say it is unfortunate that all that base reader refactoring had to happen in this PR with all these other changes. Do any of the reader documentation pages need to be updated to reflect the separate classes like the custom reader instructions?
In my defense, I didn't see this as a major refactor, as I just split a class into two without any changes to the code iirc, and renamed
We don't need to update any documentation now I think, as we (or I) am still looking for the best way to implement that kind of reader (ie reader with many files depending on each other). But in the future, we indeed need to improve the documentation to show how to create a custom reader this way. |
I think the |
to be clear |
|
Ah great. Sounds good then. |
| for key in dataset_keys: | ||
| for handler in self.storage_items.values(): | ||
| val = handler.get_dataset(key, info=dict()) | ||
| if val is not None: | ||
| val.attrs["start_time"] = handler.start_time | ||
| if key["name"] not in ["longitude", "latitude"]: | ||
| lonlats = self.load([DataID(self._id_keys, name="longitude", polarization=key["polarization"]), | ||
| DataID(self._id_keys, name="latitude", polarization=key["polarization"])]) | ||
| gcps = val.coords["spatial_ref"].attrs["gcps"] | ||
| from pyresample.future.geometry import SwathDefinition | ||
| val.attrs["area"] = SwathDefinition(lonlats["longitude"], lonlats["latitude"], | ||
| attrs=dict(gcps=gcps)) | ||
| datasets[key] = val | ||
| continue | ||
| return datasets |
There was a problem hiding this comment.
I'll merge this PR because it fixes the CI for everyone else, but it'd be nice to have a followup PR to fix CodeScene issues like this method being too complex.
This PR refactors the SAR reader not to depend on the FileYAMLReader.
Depends on pytroll/pyresample#578
Todo: