Add Sun-Earth distance corrector utility and apply in SEVIRI readers #1341

simonrp84 · 2020-09-04T10:15:59Z

The distance between the sun and the Earth varies slightly during the year due to the Earth's orbital eccentricity. This means that the incoming solar irradiance, and hence outgoing radiance measured by a satellite sensor, also varies throughout the year as a function of day.

Many sensors do not account for this in their L1 data products, so a correction must be applied.
As an example, for SEVIRI:

Reflectance = (pi * se_dist^2 * radiance) / (solar_irradiance * cos(solar_zenith))

Where:
se_dist is the Sun-Earth distance on a given day.
radiance is the satellite-measured pixel radiance.
solar_irradiance is the average solar irradiance for the band.
solar_zenith is the solar zenith angle (with nadir = 0) for a given pixel.

Currently, the SEVIRI reader (and possibly others too) doesn't apply the se_dist factor, and therefore the reflectances given by satpy are slightly incorrect (by up to ~4%).

This PR creates a utility function apply_earthsun_distance_correction that applies the correction. It also adds this function to the SEVIRI calibration code and adds the various test functions to ensure everything works.
In addition, a remove_earthsun_distance_correction function is also supplied for the case that a user wishes to remove the correction.

Closes SEVIRI reader doesn't include Earth-Sun distance in the rad->refl calibration #1334
Tests added
Passes flake8 satpy
Fully documented

…is in the solar irradiance value.

…lculating reflectances in the solar channels.

…calculating the reflectance for a given band.

coveralls · 2020-09-04T10:38:16Z

Coverage increased (+0.1%) to 90.377% when pulling 098fb6b on simonrp84:reflectance_earthsun_distance into e77cd29 on pytroll:master.

codecov · 2020-09-04T10:47:15Z

Codecov Report

Merging #1341 into master will increase coverage by 0.13%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master    #1341      +/-   ##
==========================================
+ Coverage   90.24%   90.37%   +0.13%     
==========================================
  Files         222      228       +6     
  Lines       32719    33068     +349     
==========================================
+ Hits        29526    29886     +360     
+ Misses       3193     3182      -11

Impacted Files	Coverage Δ
satpy/readers/seviri_l1b_hrit.py	`92.18% <ø> (ø)`
satpy/readers/seviri_l1b_native.py	`73.94% <ø> (ø)`
satpy/readers/seviri_l1b_nc.py	`43.10% <ø> (ø)`
satpy/readers/seviri_base.py	`92.07% <100.00%> (+1.17%)`	⬆️
satpy/readers/utils.py	`90.00% <100.00%> (+1.80%)`	⬆️
satpy/tests/reader_tests/test_seviri_base.py	`100.00% <100.00%> (ø)`
.../tests/reader_tests/test_seviri_l1b_calibration.py	`96.72% <100.00%> (+0.49%)`	⬆️
satpy/tests/reader_tests/test_utils.py	`100.00% <100.00%> (ø)`
satpy/tests/reader_tests/test_netcdf_utils.py	`94.79% <0.00%> (-0.82%)`	⬇️
satpy/composites/__init__.py	`82.76% <0.00%> (-0.17%)`	⬇️
... and 24 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e77cd29...098fb6b. Read the comment docs.

mraspaud

Looks good so far, just a couple of programmatic corrections I'd like to see included here. Otherwise, I think you missed _vis_calibrate usages in reader_tests and in seviri_l1b_nc.

satpy/readers/seviri_base.py

satpy/readers/utils.py

…ime if included in the xarray data. Otherwise, fall back to value being passed or raise an error.

mraspaud · 2020-09-04T15:19:23Z

The preferred way to access dataarray attributes is array.attrs['something']

simonrp84 · 2020-09-04T15:27:57Z

Fixed :-)

mraspaud

Now that the code works, let's refactor to make it right :)

satpy/readers/utils.py

satpy/tests/reader_tests/test_utils.py

sfinkens

Very nice feature, thanks! A couple of questions and comments.

satpy/readers/utils.py

satpy/readers/seviri_base.py

sfinkens · 2020-09-08T10:27:38Z

Also pinging @ninahakansson here, as this might be relevant in level1c4pps

ninahakansson · 2020-09-08T10:41:52Z

Thanks @sfinkens for notifying me. At the moment in level1c4pps assume that sun-earth distance correction is not applied for SEVIRI, so this need to be updated when this is merged to satpy.

…e correction.

simonrp84 · 2020-09-08T20:31:07Z

I just pushed some changes that are not yet complete, as my git skills are low and I want to transfer my code changes between home and office :-) Will complete the updates after review ASAP.

… function to retrieve UTC time. Also restructure appropriate tests into a class.

… correction is applied and, if so, what the Sun-Earth distance is.

…by the Sun-Earth distance correction.

sfinkens

LGTM

sfinkens · 2020-09-09T13:04:32Z

satpy/readers/utils.py

+    utc_date = get_array_date(reflectance, utc_date)
+    sun_earth_dist = sun_earth_distance_correction(utc_date)
+
+    reflectance.attrs['sun_earth_distance_correction_applied'] = False


@ninahakansson You can use these attributes in level1c4pps

mraspaud

LGTM

mraspaud · 2020-09-09T13:09:41Z

@simonrp84 one last thing: could you mention in the module docstrings of the seviri readers that earth-sun distance correction is now applied?

…stance correction is taken into account when computing reflectances.

simonrp84 · 2020-09-09T13:30:01Z

@mraspaud Ok, this is done now!

mraspaud · 2020-09-09T13:33:34Z

Perfect. Merging this when the tests pass.

simonrp84 added 3 commits September 4, 2020 10:21

Explicitly multiply reflectance values by pi rather than including th…

ebe68b4

…is in the solar irradiance value.

Add function to apply/remove the sun-earth distance factor used in ca…

b9afa93

…lculating reflectances in the solar channels.

Update the SEVIRI readers to account for the Sun-Earth distance when …

efd3b57

…calculating the reflectance for a given band.

simonrp84 requested review from adybbroe, djhoese, mraspaud and sfinkens as code owners September 4, 2020 10:15

simonrp84 self-assigned this Sep 4, 2020

simonrp84 added bug refactor labels Sep 4, 2020

Update SEVIRI base tests to keep deepcode happy re. array comparisons.

00b403e

mraspaud requested changes Sep 4, 2020

View reviewed changes

satpy/readers/seviri_base.py Outdated Show resolved Hide resolved

satpy/readers/utils.py Outdated Show resolved Hide resolved

simonrp84 added 2 commits September 4, 2020 13:32

Update Sun_Earth distance correction to use start_time or scheduled_t…

8d1f457

…ime if included in the xarray data. Otherwise, fall back to value being passed or raise an error.

Stylistic changes to the SEVIRI tests

660c2ae

Modify way of accessing attributes in the Sun-Earth distance correction.

ffdc703

mraspaud requested changes Sep 7, 2020

View reviewed changes

satpy/readers/utils.py Outdated Show resolved Hide resolved

satpy/tests/reader_tests/test_utils.py Outdated Show resolved Hide resolved

sfinkens requested changes Sep 8, 2020

View reviewed changes

Update utils and seviri_base for clearer use of the sun-earth distanc…

31098c6

…e correction.

simonrp84 added 2 commits September 9, 2020 10:18

Improve utility functions for Sun-Earth distance correction to create…

90b06df

… function to retrieve UTC time. Also restructure appropriate tests into a class.

Update code style to match PEP

6de1213

simonrp84 added this to the v0.23.0 milestone Sep 9, 2020

simonrp84 added 2 commits September 9, 2020 11:46

Add attribute to Sun-Earth distance correction specifying whether the…

85c8a06

… correction is applied and, if so, what the Sun-Earth distance is.

Update SEVIRI calibration tests to match the new attributes produced …

91795eb

…by the Sun-Earth distance correction.

sfinkens approved these changes Sep 9, 2020

View reviewed changes

sfinkens reviewed Sep 9, 2020

View reviewed changes

mraspaud approved these changes Sep 9, 2020

View reviewed changes

Update documentation for SEVIRI readers to note that the Earth-Sun di…

098fb6b

…stance correction is taken into account when computing reflectances.

mraspaud merged commit 0434faf into pytroll:master Sep 9, 2020

simonrp84 deleted the reflectance_earthsun_distance branch October 15, 2020 08:38

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Sun-Earth distance corrector utility and apply in SEVIRI readers #1341

Add Sun-Earth distance corrector utility and apply in SEVIRI readers #1341

simonrp84 commented Sep 4, 2020 •

edited

Loading

coveralls commented Sep 4, 2020 •

edited

Loading

codecov bot commented Sep 4, 2020 •

edited

Loading

mraspaud left a comment

mraspaud commented Sep 4, 2020

simonrp84 commented Sep 4, 2020

mraspaud left a comment •

edited

Loading

sfinkens left a comment

sfinkens commented Sep 8, 2020

ninahakansson commented Sep 8, 2020

simonrp84 commented Sep 8, 2020

sfinkens left a comment

sfinkens Sep 9, 2020

mraspaud left a comment

mraspaud commented Sep 9, 2020

simonrp84 commented Sep 9, 2020

mraspaud commented Sep 9, 2020

Add Sun-Earth distance corrector utility and apply in SEVIRI readers #1341

Add Sun-Earth distance corrector utility and apply in SEVIRI readers #1341

Conversation

simonrp84 commented Sep 4, 2020 • edited Loading

coveralls commented Sep 4, 2020 • edited Loading

codecov bot commented Sep 4, 2020 • edited Loading

Codecov Report

mraspaud left a comment

Choose a reason for hiding this comment

mraspaud commented Sep 4, 2020

simonrp84 commented Sep 4, 2020

mraspaud left a comment • edited Loading

Choose a reason for hiding this comment

sfinkens left a comment

Choose a reason for hiding this comment

sfinkens commented Sep 8, 2020

ninahakansson commented Sep 8, 2020

simonrp84 commented Sep 8, 2020

sfinkens left a comment

Choose a reason for hiding this comment

sfinkens Sep 9, 2020

Choose a reason for hiding this comment

mraspaud left a comment

Choose a reason for hiding this comment

mraspaud commented Sep 9, 2020

simonrp84 commented Sep 9, 2020

mraspaud commented Sep 9, 2020

simonrp84 commented Sep 4, 2020 •

edited

Loading

coveralls commented Sep 4, 2020 •

edited

Loading

codecov bot commented Sep 4, 2020 •

edited

Loading

mraspaud left a comment •

edited

Loading